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Foreword 


The term "design review" has found a rather wide use in recent years both in 
aerospace industry and in Government. However, depending on the user or context 
of use, the term is applied to a fairly wide range of scopes and kinds of activity. 
NASA Reliability Publication NPC 250-1, entitled "Reliability Program Provisions 
for Space System Contractors," prescribes design review as one of the activities 
necessary to the development of NASA hardware. Although the language in NPC 
250-1 is explicit in describing the basic purpose of the design review task and a 
number of specific contractor and Government actions in support of the require- 
ment, the scope and purpose of NPC 250-1 do not permit a full illustration of the 
range of functions by which design review should serve a hardware development 
program (a) at various levels of assembly, (b) at various milestones in the develop- 
ment process, and (c) in projects of differing size and significance. 

The present publication has been structured to describe the design review activ- 
ity, not only in its overall aspect but also in its operation at each of the various 
hardware levels, program milestones, and project types. The approach is to pro- 
vide a generic and functional description without expressing any procedural prefer- 
ences except on grounds of effectiveness in accomplishing the purpose of the review 
activity. 

It is the objective of this publication to serve as an information tool to assist 
NASA personnel and installations in communicating with their contractors on design 
review. The procedures and elements of reviews covered here are intended to be 
descriptive and not mandatory. 

The principal author of this document is R. E. Boss, who was assisted by 
C. H. Me Gaff in, both of the Martin Company, and the effort has been guided and 
edited by D. S. Liberman of this office. However, significant assistance has been 
provided by all NASA field installations and Headquarters program offices in con- 
structively reviewing and commenting on the drafts, and this is gratefully ac- 
knowledged. 

Although developed under contract (NASw-1128), this publication incorporates 
many comments reflecting NASA experience beyond that of the contractor. The 
result, therefore, reflects a considerable breadth of engineering management 
experience by NASA and its contractors. 


John E. Condon, Director 
Reliability and Quality Assurance Office 
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Summary 


This present publication is intended to serve as an aid for instructing technically trained per- 
sonnel in the implementation and evaluation of a design review program. It describes in detail 
those reviews performed by the contractor with customer participation and touches lightly on the 
various reviews and buy-offs conducted by the customer. 

A design review program is a systematized and disciplined application of the broad technical 
competence of the contractor and the customer to a product. It is the only technique for independ- 
ent evaluation of a product prior to the testing phase and is a necessary technique for the achieve- 
ment of reliability. The basic objective of a review program is to provide assurance to customer 
and contractor management, by formalized documentation of the decision logic, that the design 
satisfies the program requirements. Detailed objectives of the design review (from ref. 1) are: 

(1) To evaluate the capability of the design to meet the total system requirements 

(2) To determine the effects of procurement, assembly test, shipping, storage, human fac- 
tors , and maintenance on the achievement of the design goals 

(3) To identify problems of process control, production, and parts and material procurement 

(4) To consider the effects op performance of proposed configuration changes 

(5) To evaluate the operational functions of the design with respect to the current known 
state-of-the-art 

(6) To evaluate the adequacy of the specification to meet the intended operational use 

(7) To determine whether the design conforms with the specification requirements 

(8) To provide for optimization of design within the functional performance requirements 

The elements of a design review program are: 

(1) Review -team makeup 

(2) Responsibility and authority of chairman and team 

(3) Customer participation 

(4) Review frequency 

(5) Data inputs 

(6) Data outputs 

(7) Continuity and followup 

To satisfy the objectives of the design review, the review team must have a collective techni- 
cal competency greater than that of the designer(s) of the design being reviewed. There are 
several resources for achieving this competency: 

(1) Permanent reviewers 

(2) Recruited specialists 

(3) Staff or consultant authorities 

(4) Various combinations of the above 

Selection of the team makeup for the various levels of review should be based on the advantages of 
each approach discussed in this publication. 
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It is the responsibility of the design review activity to direct program management attention 
to design deficiencies while correction is still chronologically and economically feasible. The 
review team may, or may not, have directive authority; however, in any case the review docu- 
mentation must not be censored, whether the recommendations are accepted or not. 

The participation of Government representatives in design reviews is both necessary and de- 
sirable; it is necessary since one objective of the review is to assure the customer that the design 
will satisfy the system requirements, and desirable since the customer will have inputs not other- 
wise available to the contractor. Contractual implications of directions by the Government repre- 
sentative usually require separate discussion, since the design review team will not include 
specialists in that field. 

The scheduling of a design review program properly related to program significance, system 
complexity, and design milestones is probably the most difficult of the program elements. Fac- 
tors to be considered are: 

(1) Program significance 

(2) Complexity 

(3) Technical background 

(4) Cost 

Reviews may be grouped into milestone categories generally applicable to any design and develop- 
ment effort. These are: 

(1) Preliminary design review at system, subsystem, and component level 

(2) Prepackaging review, for electrical and electromechanical components 

(3) Prerelease review, for all components prior to manufacture, also at subsystem level by 
concurrent reviews at component level 

(4) Acceptance reviews, for all levels 

(5) Special purpose reviews and buy-offs 

Data provided to the review team prior to the review will vary in specifics with the particular 
milestones but in general must define the item being reviewed, give the requirements of the item, 
and describe its interfaces with other items. 

Documentation of the design review must record the team makeup, the review level, the input 
material, the decision items (not just action items), and the decision logic that established the 
validity of subsequent corrective action as well as the confidence that management (contractor and 
customer) assigns to the product. 

Continuity in a design review program is particularly difficult to achieve and depends to a 
large extent on the documentation of previous reviews. Current reviews must avoid rehashing the 
old problems. Followup is necessary to make certain that appropriate design change action has 
been taken or that additional study has validated the original design. 

Application of the precepts of this publication will result in a reduction of end costs of any 
program through prudent and timely investment of manpower to reduce the larger cost risks of 
subsequent failure. However, for planning purposes the apparent cost of a design review pro- 
gram properly related to program significance and complexity will be between 1 and 2 percent of 
the total engineering effort. 


CHAPTER 1 


Definition and Purpose 

NASA Reliability Publication NPC 250-1, entitled "Reliability Program Provisions for Space 
System Contractors” (ref. 2), states that the contractor should establish and conduct a formal 
program of planned, scheduled, and documented design reviews at the system, subsystem, and 
major component level. 

A design review program is a systematized and disciplined application of the broad technical 
competence of the contractor and the customer to a product. Its purpose is to improve the product 
where necessary and to provide assurance to contractor and customer management, by formalized 
documentation of the decision logic, that the most satisfactory design has been selected to meet 
the program requirements. In creating this program requirement, NASA has expressed the need 
for additional control and visibility in the development process. 

The primary purpose of the present publication is to describe in further detail for technically 
trained personnel the various functional elements of a design review program as called for in 
NPC 250-1 (ref. 2). This present publication also describes generically several of the various 
types of customer reviews and buy-offs which are used to verify the adequacy of the design and 
the hardware in program phases subsequent to design (and sometimes even during the design 
phases) . 

Because of the wide diversity of NASA programs, which range from unmanned probes evalu- 
ating single facets of the space environment to the bulk transportation problem associated with 
manned interplanetary travel, there is room for many interpretations of the design review proc- 
ess. However, the design review function, no matter how applied, should have certain attributes 
in order to be effective: 

(1) It must provide an objective examination of the design by persons other than the designer. 

(2) It must be conducted by a limited number of competent engineers, each with a compe- 
tency at least equal to that of the designer in his own technical specialty but still suffi- 
ciently practical to maintain communication with other participants. 

(3) It must serve to discipline the process of documenting the design decision logic. 

(4) It must foster an atmosphere of free discussion of the designers conclusions and the 
process by which he reached them. It must never take on the atmosphere of a board of 
inquiry whose object is to put the designer on the defensive. 

(5) It must include provisions for effective review of subcontractor designed items. 

The scope of the design review program will normally be defined contractually in the reliabil- 
ity program plan; however, this does not imply that the conduct of the design review program is a 
reliability function. 

This publication describes a design review program conducted by the contractor with custom- 
er participation. In this type of program there are no customer conducted reviews prior to the 
buy-off activities of acceptance review, special readiness reviews, and flight readiness review. 
However, it should be noted that there are also programs, particularly in the manned space area, 
where the customer conducts most of the major reviews even during the design phase, although 
the contractor makes the review presentations. Reviews of this type, while generally analogous 
to the preliminary, prepackaging, and prerelease reviews described here, will differ in that the 
review team usually has directive authority (via appropriate contractual arrangements) and the 
reviews have some characteristics of an intermediate customer buy-off. 
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The following chapters discuss the design review process from several aspects. Chapter 2 
covers the basic elements of the review process including the reviewers and their responsibility, 
the scheduling of a review program, data inputs and outputs, and followup on recommendations. 
Chapter 3 discusses the technique of review as applied to successive program milestones and 
levels of hardware; it covers not only true design reviews but also some buy-off activities. 
Finally, Chapter 4 briefly treats the proper scope of review programs for different projects, and 
Chapter 5 shows the costs of a design review program as applied to three typical programs of 
differing magnitudes. 

For the reader's aid, a list of definitions of terms and abbreviations used in this publication 
is given in Appendix A. Also, a list of publications of further interest is given in the 
Bibliography. 


CHAPTER 2 


Elements of Design Review 

A design review program consists of a series of reviews at various hardware levels at suc- 
cessive milestones in the life of the project. However, the approach to the program in terms of 
seven basic elements can have a considerable influence on the overall effectiveness. These ele- 
ments are: 

(1) Review-team makeup 

(2) Responsibility and authority of chairman and team 

(3) Customer participation 

(4) Review frequency 

(5) Data inputs 

(6) Data outputs 

(7) Continuity and followup 

Each of these elements and its importance to the program are discussed in the present chapter. 

REVIEW TEAM 
General 

In order to satisfy the objective of the design review, the review team must have a collective 
technical competency greater than that of the designer(s) of the design being reviewed. If this 
collective advantage does not exist, the program will not satisfy the objectives. (The designer 
will be placed in a position of having to lead the review, and, in a sense, interrogate himself.) 

Team Compositions 

There are various resources for achieving this technical competency. The principal ones 

are: 

(1) Permanent reviewers 

(2) Recruited specialists 

(3) Staff or consultant authorities 

(4) Combinations of above team compositions 

Permanent Reviewers 

A permanent review-team nucleus can be established with manning broad enough to handle all 
levels of review. The basic composition can be senior staff-level engineers of the contractor or- 
ganization. The individuals need not necessarily be the "number one" men in their respective 
specialties, since use of their collective capabilities and disciplines will satisfy the review pro- 
gram objectives. Of course, they must have recognized capabilities in the specialties. This 
nucleus will contribute a cohesion and directional force not available with other techniques. 

Recruited Specialists 

Specialists can be gathered from other programs and staffs on demand. These persons, by 
virtue of the variety of their specialized viewpoints and isolation from the pressures of schedule 
and prior commitments, can supply knowledgeable unbiased criticism of the design. These 
teams will be high in individual competency but weak in preparation and motivational force . 
TTtilization of these specialists subjects the design to perceptive scrutiny which may well bring to 
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light areas for improvement in performance or reliability. However, it is difficult for personnel 
with no previous contact with a program to evaluate properly the equally important trade-offs of 
cost, schedule, customer desires, etc. 

Staff or Consultant Authorities 

Technical authorities, either staff or consultant, can be utilized to achieve the required col- 
lective competency. Here again the design benefits from a knowledgeable examination with appli- 
cation of state-of-the-art principles. However, the inputs of the authority must be limited to his 
area of specialty in order to avoid suggestions for changes which are too extravagant or subjec- 
tive to be accepted by management (whereas limited objective changes might be accepted). The 
review effort should not result in design improvement without sufficient cause. 

Combinations of Above Review-Team Compositions 

Combinations of the above personnel depend on the circumstances of the contractor and the 
item under review. Where circumstances dictate the combination approach, the selection of 
teams for various levels of review should be based on the relative advantages discussed below. 

Summary 

The team compositions discussed above have their strengths and weaknesses when applied to 
var’*^ 1 levels of review. In general, the inputs of technical authorities are more useful in the 
conceptual stages, where freedom of thought is not restricted by cost of implementation (hard- 
ware, test, and paperwork revisions). However, corrective measures take on a more practical 
aspect as the design progresses into the detailed phases. Here the authority might be consulted 
in the face of apparently insurmountable problems , but for the most part detail problems can be 
resolved and the design crystallized by application of basic design principles by a broadbased, 
hardware -oriented review team. 

Regardless of the technique used, the review team must have sufficient breadth to handle the 
several aspects of the review item (performance, reliability, cost maintainability, etc.) and the 
interfaces and interactions with adjacent items. 

RESPONSIBILITY AND AUTHORITY OF CHAIRMAN AND TEAM 

It is the responsibility of the design review activity to direct program management attention 
to design deficiencies while correction is still chronologically and economically feasible. The 
following discussion treats specific obligations of the design review participants. 

Management has the responsibility for providing a favorable climate for the design review 
program including assignment of personnel, time for preparation, and so forth. This can best be 
accomplished by including a specific item in project cost negotiations for design review at the out- 
set of the program. The reliability program plan should include a comprehensive schedule of the 
needed reviews relative to design milestones. The reliability organization has the responsibility 
for including the coverage in the plan. 

The review-team chairman must define the input data package, set the agenda, and document 
the meeting; the coordinator must bring about the preparatory activities, secure the services of 
required outside personnel, and inform the customer of review-team activities. 

The review team has a responsibility to prepare for the review by utilizing fully the data given 
them. A review that degenerates into a low-level education program will be wasted. 

Although the members of the review team must have authority to call for additional technical 
competency (specialists, consultants, etc.) or additional data, they must not transfer their re- 
sponsibility to the technical "expert." His testimony must be considered as only one input to be 
balanced against other significant factors (program impact, necessity, alternate designs, etc.). 
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Review-team personnel must accept responsibility for the pertinency and adequacy of the 
team output, and project management must assume that the output is a responsible attempt to 
fulfill the review objectives. This is necessary in order to create the proper atmosphere for 
conscientious design evaluation even though the responsibilities of the review team stop short of 
accomplishment of the recommended improvements. The review team may or may not have di- 
rective authority for initiation of corrective action. Where it does not, its function is advisory to 
the contractor program manager rather than directive; his office, in turn, triggers the correc- 
tive action process if the recommended action is approved. If the recommendations are not 
accepted the review documentation must not be censored; the logic of rejection itself must be 
documented. It is the responsibility of the review-team chairman to ensure that this has been 
done by management. 

In some cases the review team does in effect have directive authority (such as where the 
project manager or his immediate deputies act as chairman) . Here, trade-off decisions may be 
made in the review itself, wherein prime candidate solutions to design problems may be rejected 
because of program constraints other than technical ones. In such situations, the review report 
must document both the proposed solution and the logic of rejection. 

The selection of the review chairman and assignment of his duties are management preroga- 
tives dependent on many variables, such as availability of personnel, review level, and program 
significance. Some organizations use permanent chairmen while others may use technical de- 
partment heads or staff engineers. Duties of the chairman as they relate to specific reviews will 
be discussed more fully in subsequent chapters. The value of a specific review will depend 
greatly on the manner in which the chairman carries out these duties. 

The necessity for complete objectivity (i.e. , the ability to criticize the design without injec- 
tion of personal or technical prejudice) on the part of the design review team is frequently men- 
tioned; however, the design review effectiveness can be as severely restricted by unconstrained 
objectivity as by lack of objectivity. On the one hand, lack of any degree of objectivity, that is, a 
tendency to perform design review completely within the confines of program constraints (includ- 
ing schedules, costs, etc.), subjects the mental processes of the review team to the same 
restrictions as those under which the designer(s) has been functioning. A good technical solution 
to a design weakness may be ignored in consideration of an unduly restrictive concept of program 
constraints. On the other hand, the reviewer must have some realistic concept of program con- 
straints; and unless he attempts to remain within this concept, he is apt to propose extravagant 
or unnecessary revisions. The management rejection prerogative may prevent serious damage to 
the program effort; however, the review functions of design improvement and assurance are side- 
tracked in the process. A marginal design is not improved when an extravagant revision is re- 
jected, and the assurance of adequacy of the existing design is lessened when an unnecessary 
change is proposed. Furthermore, management’s confidence in the review effort decreases with 
increase in impracticality of the outputs . 

It is the responsibility of the review team to initiate inquiries into areas that appear techni- 
cally weak and to introduce design alternatives for discussion. Objectivity tempered with the re- 
viewer’s concept of practical restraints is the key to a successful design review. 

CUSTOMER PARTICIPATION 

The participation of the customer (i.e. , Government representatives) in design reviews is 
both necessary and desirable: necessary because one reason for design reviews is to assure the 
customer that the design will satisfy system objectives, and desirable because the customer will 
frequently have input information not otherwise available to the contractor which can significantly 
improve design and reliability or reduce the program cost. Their participation is particularly 
fruitful during the conceptual period, when associated customer experience and detailed know- 
ledge of related projects can contribute to the establishment of sound basic premises. The 
Government representative should make known the necessity of any predetermined course which 
is based on constraints that may not have been available to the contractor. 
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The basic philosophy of design review is conveyed to the contractor by NPC 250-1 (ref. 2). 

It is incorporated into the reliability program plan and agreed upon at the start of the program. 
However, no matter how specific they may be, these requirements are still subject to interpreta- 
tion in application. Coordination between customer and contractor during the initial working 
meetings affords an opportunity to clarify most differences of interpretation. 

The attending Government representative must be subject to the same disciplines of prepara- 
tion as those of the contractor personnel. Fifteen days prior to review (NPC 250-1, par. 3.6. l,b) 
the contractor will identify the review element, confirm the date and location, and present a data 
package. Like all attendees, the Government representatives must thoroughly familiarize them- 
selves with the data package and be prepared to substantiate their views prior to the meeting. 

The 15-day notification may prove difficult in highly complex systems, but such systems will nor- 
mally have an office for a resident Government representative at the contractor's facility, and 
this should facilitate communication and transmittal of information. A mutually agreeable reduc- 
tion in notification time may be feasible in view of the complexity of the system and the extent to 
which the customer has been involved in the day-to-day design. 

The contractual implications of directions from the Government representative will usually 
be determined by contracting and legal personnel, neither of whom is present at the review. 
Therefore, the in-scope/out-of-scope argument should not be injected into the review meeting. 
Normally, "scope” agreements generated at the design review are not binding, and the contractor 
has some specified time (as little as 5 days on some programs) to request contractual coverage. 

Because the design review necessarily involves a critical evaluation of all aspects of the 
design, the conscientious contractor will frequently expose internal shortcomings to the collective 
criticism of the group. All members of the group should be most careful to use such information 
properly and objectively so as to prevent any reduction in its flow which would reduce the effec- 
tiveness of subsequent reviews. The ability of the participating Government representative to 
convince the contractor’s personnel of his objectivity and discretion, particularly in this regard, 
can have a considerable influence on the effectiveness of the design review program. 

REVIEW FREQUENCY 

The scheduling of a design review program properly related to program significance, system 
complexity, and design milestones is probably the most difficult of the review program elements. 
The schedule provides the basis for both planning and estimation of manpower and costs and for 
evaluation of adequacy of the program. In this section, we will discuss the effect of several vari- 
ables on the scheduling of reviews and the relation between the review program and the milestones 
of the design and development program. Subsequent chapters will demonstrate the applicability to 
representative design activities. 


Influence of Project Characteristics 


Program Significance 

Any space-system program should be designed and developed to have a high probability of 
success, but we are willing to take more risk in some cases than in others depending on the pro- 
gram significance. The success of a program ranking high in the list of national objectives, such 
as a manned space program, is extremely significant, and the design and development phase will 
be scrutinized very closely. 

Complexity 

The complexity of the system also will determine the frequency of reviews. A complex sys- 
tem will require more reviews than a simple system, not only because of the larger number of 
system elements, but also because the interfaces become more numerous and more subtle. 
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Previously Proven Components 

Use of previously proven designs in a new system can significantly affect the number of re- 
views required. In this case it is necessary to assure that the new usage is no more demanding 
in either performance or environment than the old and that the new interfaces are compatible with 
the equipment. The schedules associated with integration of those units into the new design are 
somewhat flexible. The advanced status of the development of those units permits some smooth- 
ing of the design and review workload with respect to time. However, prolonged delays should be 
avoided, since there is always the possibility that an unforeseen application problem might re- 
quire a major redesign, and what was at first a flexible schedule might suddenly become a PERT 
critical path. Similarly, prior use does not assure availability of complete documentation of that 
usage, particularly when that data must cross company lines. A good initial step is to obtain and 
evaluate the supporting qualification test data for conformance to the standards and environmental 
levels of the new program. 

Cost of Review Program 

Cost or, more precisely, lack of money is seldom a valid reason (although frequently used) 
for decrease in the review frequency. Design review is a necessary technique for the achieve- 
ment of a reliable design. Frequency of review will be dictated by the factors of significance and 
complexity. 

Influence of Hardware Levels and Development Milestones 

Once the frequency of the design reviews has been established on the basis of the above 
factors , the specific reviews relative to the system subdivisions and the design and develop- 
ment milestones can be defined. This definition is generally bound by standardized engineer- 
ing practices; selection of hardware subdivisions for review is guided by the tecnhologies 
involved, and selection of the project milestones for reviews is governed by the sequence of 
steps in design normal for each of the various hardware subdivision levels and types. 

Selection of system subdivisions usually follows the established levels of systems, sub- 
systems, components, and parts. A system, as defined by Appendix A of NASA NPC 250-1 (ref. 
2), is ’’one of the principal functioning entities comprising the project hardware and related oper- 
ational services within a project or flight mission. Ordinarily, a system is the first major sub- 
division of project work. Similarly, a subsystem is a major functioning entity within a system." 
Thus, for NASA purposes, the system level is associated with the operational function, or mis- 
sion, of a space activity. The subsystem level divides the overall activity into those technologies 
required to perform or support the mission. Representative examples include power and distri- 
bution (electrical), flight controls and guidance (electromechanical), and propulsion (mechanical). 
The next level of subdivision] is the component. NPC 250-1 defines a component as "A combina- 
tion of parts, subassemblies or assemblies, usually self-contained, which performs a distinctive 
function in the operation of the overall equipment," i.e. , a "black box." Design review, as con- 
sidered here, is not carried below the level of component. However, consideration of the appli- 
cation of parts and materials in the components, although detailed and time-consuming, is basic 
to the review of the component design. NPC 250-1, paragraph 3.9.6, requires specific reviews 
covering the application of parts and materials; the outputs of these reviews are an essential 
input to component design review. Another as yet unpublished manual of this series discusses 
those reviews in detail. 

Reviews may be grouped into milestone categories which are generally applicable to any 
design and development effort. These categories are: 

(1) The preliminary design review, sometimes called conceptual review 

(2) The prepackaging review — prior to preparation of detail drawings 

(3) The prerelease review— prior to release for qualification test unit production 

(4) The postqualification review— immediately following qualification and preferably, but not 
usually, before start of production 
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(5) Acceptance review— prior to delivery to the customer 

(6) Flight readiness reviews— following completion of ground testing 

It should be noted that reviews (1), (2), and (3) above are design reviews in the purest sense of 
the word; usually they are conducted by the contractor with customer participation. Review (4) is 
functionally a private readiness review by the contractor to assess the readiness of the system 
for presentation to the customer for acceptance. Finally, reviews (5) and (6) are essentially buy- 
offs conducted by the customer with contractor participation. 

It should be noted that the review categories above are selected by technical milestones and 
not by technical specialties. Formal reviews by specialty (e.g. , reliability review and value re- 
view) are not advisable, since the objective of the review is to consider the trade-offs, compro- 
mises, and controversies. A system, subsystem, or component can always be made either more 
reliable or cheaper, but not always both. The proper trade-off cannot be appraised objectively 
by the concerned specialty. The needed perspective can only be achieved through joint evaluation 
by all concerned specialties. 

Figure 1 presents a generalized milestone chart of a design and development effort such as 
might be encountered by a space-system contractor. The chart shows the completion points 
(milestones) of significant portions of the effort. Each point in the conceptual and component de- 
sign stages is accompanied by a letter designation corresponding to the above listing of review 
categories to indicate the type of review to be conducted at that point. Review frequency should 
decrease measurably as hardware production commences since the basic job of design verifica- 
tion and trade-off through review must be completed by this stage for maximum effectiveness. 
Subsequent reviews, conducted at the system level, emphasize the assurance aspect, as implied 
by their names— acceptance review and flight readiness review. 

The chart shows only a single sample of each milestone, whereas an actual project may be 
composed of any number of related subsystems and these in turn may contain numerous compo- 
nents. The chart is therefore a composite of numerous overlaid milestones (except for the 
system -concept milestone at the left and the ”first article” milestones at the upper right). 

Aside from the knowledge flow indicated on the chart, chronological independence exists. 

Unless multiple paths of flow exist (i.e. , several concurrent activities are required) from one 
milestone to another, the efforts to achieve one milestone are entirely independent of those to achieve 
another and may be conducted concurrently or months apart; program schedules will determine 
procedure. For instance, the generation of circuitry for an electrical component is independent 
of the detail design of the structural subassembly into which the component will be installed. 
Similarly, the detail development of one component is independent of a second, since the interface 
data have previously been established in the subsystem conceptual stage. The parallel but time- 
independent efforts are represented by overlaid "component detail” milestones. 

Frequently, a decision is made to permit full production prior to completion of qualification 
testing, as indicated by the dotted line around the test. That decision assumes acceptance of a 
risk of subsequent changes in order to improve schedule or reduce setup costs. 

DATA INPUTS 

Data provided to the review team prior to the review will vary in specifics depending on the 
milestone at which the review is being conducted, but in general they must define the item being 
reviewed, give its requirements, and describe its interfaces with other items. These data might 
normally be thought of as drawing(s), specif ication(s) , and interface specification(s) . For exam- 
ple, a component review of an in-house built item might require: 

(1) Detail drawing (pictorial representation, descriptions of required materials, finish, di- 
mensions, tolerances, fabrication, and assembly instructions, etc.) 
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Figure 3. —Milestone chart of a system design and development effort, a, Preliminary design review; b, prepackaging design 
review; c, prerelease design review; d, postqualification design review; e, acceptance review; f, flight readiness review; 
g, subsystem status review. 
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(2) Installation drawing (general configuration, attaching hardware, and information to lo- 
cate, position, and mount the item relative to a fixed reference) 

(3) Circuit schematic diagram (function symbols with interconnections to illustrate circuit 
operation) 

(4) Component specification (functional characteristics and test requirements to verify these 
parameters) 

(5) Parts and materials application review data (detail reviews) 

(6) Subsystem (or system) specification (for interface functional characteristics and test 
requirements) 

(7) System design data report (system description and specific design requirements such as 
space and weight considerations, mounting requirements, special environments, design 
and checkout requirements, and maintenance provisions) 

(8) System design criteria report (general design philosophy and ground rules) 

(9) Reliability analyses and failure modes/effects analyses 

The last four documents provide interface information and should reflect the latest mission 
requirements. A task in the review effort will be to verify that all mission changes have been 
implemented and that the component requirements have been reevaluated in terms of the changes. 
The important result of such a reevaluation of component requirements is that of assurance— the 
assurance that the design is capable of performing any of these new tasks under possibly increased 
environmental stresses. Assurance is also obtained that major design simplifications have been 
accomplished where possible to take advantage of associated reliability and cost benefits. This 
discussion is included in this section on data inputs (rather than that on data outputs) since the 
proposed evaluation of mission changes should be done in the preparatory phase rather than dur- 
ing the meeting. Applying any portion of meeting effort to obsolete design criteria is then 
avoided. 

Subcontractor items should receive similar consideration except that the effort is usually di- 
vided into two phases, one at the contractor facility and a followup at the vendor facility. The 
initial phase includes review of interface and installation documents as described above to confirm 
the accuracy of the requirements in the component (or procurement) specification. The procure- 
ment specification is usually expanded to include not only performance requirements but some 
component details such as external dimensions, finish, mounting surfaces, and simplified sche- 
matics with the specific detail being left to internal vendor drawings. The second phase compares 
internal vendor documentation with the procurement specification. 

In addition to the specific documents mentioned above, any review requires other more gener- 
al types of information for design review data inputs. Documented results of prior reviews with 
management approval or disapproval and followup action summaries provide a basis for current 
discussions. Also, the cognizant designer(s) must bring to the review all pertinent backup data, 
for example, design and laboratory notebooks, test reports, analyses, and results of parts and 
materials application reviews. Similarly, the reviewer should be prepared to support with data 
his initial positions (i.e. , questions on adequacy of the design). 

DATA OUTPUTS 

Basically, the data outputs of the design review activity must document: (1) the basis on 
which management (customer and contractor) has placed confidence in the product and (2) the 
logic that established need (or lack of need) for subsequent corrective action. 

The usual listing of action items is inadequate by itself since the logic of rejection of other 
input recommendations may be more significant. Design review is basically a management 
decision-making tool, and management interest at a later date may center on one of the "no- 
action” items. The recorded logic of rejection of that item by the review team will assist in 
evaluating the new information. The same reasoning applies to later review efforts. 
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Similarly, a check list is of relatively little value unless it is used almost solely as a re- 
minder device. A checked-off list contains none of the comments and considerations which lead 
to meeting conclusions and should certainly not be the sole output of the review. 

The output documentation must record the team makeup, the review level, the input material, 
the decision items (not just action items) , and the decision logic where it is not evident. It must 
be of sufficient depth to serve as a meaningful input to subsequent reviews and to management for 
approval of recommended action. 

The report should have the concurrence of all meeting attendees. It is advisable that the re- 
port be generated as the meeting progresses with resolution of each item being secured prior to 
proceeding. Although this may appear at first glance to be prohibitively time-consuming, the ad- 
vantages usually outweigh the inconvenience. Advantages include: 

(1) Added incentive for preparatory phase . Prior research and written conclusions are 
more apt to receive recognition than an educated guess. 

(2) Added directional control of meeting . Many hours can be consumed in a design review 
program by circuitous rambling. The chairman has a valuable tool in immediate docu- 
mentation because it tends to keep the meeting objectives in focus. By rephrasing dis- 
cussion thoughts into wording suitable for the report, he continually directs attention to 
the need for applicable, not extraneous, data. 

(3) More accurate recording of consensus . Postmeeting documentation is dependent on one 
person’s interpretation of meeting conclusions and usually involves some delay in being 
generated. Both conditions permit some distortion of the recorded logic. Conversely, 
on-the-spot composition secures concerted efforts at accuracy, particularly from the 
proponent of the particular idea under discussion; and the thought is captured while it is 
fresh in everyone’s mind. 

(4) Timely corrective action promoted . If concerted agreement is secured point -by -point 
on the spot, the major delays resulting from later disagreement with the accuracy of the 
recorded version of the meeting are minimized. Some variations of this technique may 
be employed to further accelerate publication of meeting results. Whatever the proce- 
dure, a timely and accurate report sets the pace for management and corrective action 
and clears the way for preparation for subsequent reviews. 

If it is not considered feasible to prepare the report during the meeting, then, as a minimum, 
a summary upon which there is concurrence must be written prior to breakup of the meeting. 

This summary will serve as the basis for the subsequent report. 

CONTINUITY AND FOLLOWUP 

If we are to realize the potentials of the design review program, continuity must be main- 
tained from meeting to meeting, and followup must be ensured until corrective action has been 
taken. Where valid design changes have been recommended and management has concurred in 
them, sufficient information must be carried over to successive reviews to avert repeated cover- 
age of these same problems or to avert significant loss of insight. If continuity and followup are 
achieved, the next review effort can be immediately directed at new facets of the item. 

Continuity is particularly difficult to achieve. Documentation provides a degree of continuity 
but is usually not complete enough in itself to stop subsequent discussions and misunderstandings. 
Complete personnel continuity is neither practical nor profitable. Practically, the same person- 
nel are seldom available for repeated review -team assignments over a period of time, and, even 
if they were available, these people are seldom capable of handling all levels of review. Possibly 
a permanent chairman can lead all reviews on a given subsystem and its components. In any 
case, new reviews must avoid unnecessary discussion of the old problems. 
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Followup is necessary to assure that the benefits really accrue. A closed-loop system is re- 
quired to make certain that appropriate design-change action has been taken or that additional 
study has validated the original design. One current system injects the action items directly into 
the hardware corrective action process and utilizes the existing followup mechanism to assure 
the same closed-loop management scrutiny of corrective action that hardware problems and fixes 
receive. In any case, whether the existing loop is used or a second directive system is generated, 
the followup that is prescribed by that procedure must become effective so that an exhaustively 
detailed followup by the design review team can be avoided. 



CHAPTER 3 


Review Categories 

The present chapter c overs the different categories of reviews identified with the program 
milestones at which they occur. These are: 

Preliminary design reviews 
Prepackaging (or breadboard) design reviews 
Prerelease design reviews 
Acceptance reviews 
Special-purpose reviews 

- Flight readiness 

- System readiness 

- Postqualification 

- Subsystem status 

- Functional 

- Failure 

The first three of these, which are truly design reviews, and the acceptance review are dis- 
cussed in detail; also included are the hardware levels to which they apply and review elements. 
The six types of special-purpose reviews, which include both special reviews and buy-offs, are 
treated more briefly. 

PRELIMINARY DESIGN REVIEW 

The preliminary design reviews (PDR or conceptual reviews) are a series of reviews at sys- 
tem, subsystem, and component level which are intended to assure contractor management and the 
customer that the proposed solutions satisfy the mission requirements; that they are within exist- 
ing technologies; that manufacturing and test facilities are available in timely fashion; and that 
contractor personnel are technically qualified, or, conversely, that subcontracts are required. 

On the basis of results of preliminary design reviews, design requirements (specifications) are 
established. 

These reviews require the concentrated effort of a broad cross section of personnel and may 
well result in major redirection of program effort. The preparatory phase of the preliminary re- 
view normally requires evaluation of major trade-off studies, as, for example, mission support 
equipment interfaces. Review findings may indicate the need for parallel development programs 
to assure the availability of an adequate design. In any case, the preliminary reviews must be 
complete evaluations of existing concepts in order to satisfy the management (customer and con- 
tractor) assurance requirement. 


System Level 


Purpose and Scope 

Preliminary review at the system level is intended to evaluate the basic engineering approach 
to solution of the mission problem before considerable effort has been expended in the subsystem 
definition phase, A system design review is frequently performed by the customer in his selec- 
tion of contractor or in his evaluation of the study phase. However, the review referred to in this 
text is a contractor effort which is conducted after mission objectives have been broken down and 
translated into subsystem requirements. By this time, system criteria are defined in terms of 
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specific mission tasks, functional requirements, and environments. Launch-vehicle or space- 
craft structure is outlined in general terms of size, weight, modular breakdown, and design 
philosophy. Major analyses normally have been completed in these areas, and, as a result, pre- 
liminary contractual specifications and design criteria have been established. Initial component 
requirements have been outlined in sufficient detail to ascertain contractor development capability 
or to determine that subcontracts are necessary . 

The initial system design approaches to fulfilling mission objectives must be reviewed against 
ail criteria thus far established, with the basic emphasis on completeness of coverage. To satisfy 
mission objectives, there are functions to be performed; equipment to perform these functions; 
tasks to be handled by each equipment; support equipment to manufacture, assemble, test, trans- 
port, and launch the flight system; facilities and personnel for each phase of operation; and so 
forth. The system design approaches must include all facets of mission operations applicable to 
the contractor and must be outlined in sufficient detail to define subsystem requirements. Com- 
plete coverage of these equipment requirements, including compatibility with customer defined 
interfaces of other mission systems, must be assured. 

Review Team 

Preparation for a preliminary system-level review requires the participation of a large 
cross section of personnel, with practically every project member contributing to the data input 
package. However, the formal review is performed by key personnel of both contractor and 
customer. 

The contractor’s chief engineer (in a large organization, the chief engineer of that division 
responsible for the project) usually performs the function of chairman. The chief engineer is the 
person in charge of all technical activity in the organization, whether he is called director of en- 
gineering, vice-president of engineering (as in small companies), or some other title. He selects 
the basic review team from among staff engineers in his own organization in order to assemble as 
well-rounded and coordinated a team nucleus as he can muster. Note that the two required char- 
acteristics of this nucleus are breadth of experience and an ability to communicate ideas . The 
basic preliminary review team will seldom be self-sufficient and must have the breadth to recog- 
nize the need for additional specialist opinions. 

Although a case could be made for inclusion of representatives of each of the engineering 
specialties contributing to the system development, the resulting review would be completely un- 
wieldy. Normally the reliability and human factors engineers are expected to cover the assur- 
ance and user aspects, while requirements for other specialized aid must be recognized by the 
review team. (See tables 1, 2, and 3 for team makeups for preliminary system reviews of var- 
ious types of projects.) 

Customer representation is essential during this review, in view of the far-reaching implica- 
tions of the decisions. Trade-off study recommendations, for example, might involve the design 
philosophy for an entire subsystem and a firm decision to act on the study recommendations re- 
quires customer concurrence. In extreme cases, information may prove inadequate for resolu- 
tion, and additional study or parallel development using each technique will be needed. In 
addition to the need for his concurrence with decisions, customer participation is valuable from 
an information -input viewpoint, since the customer has access to advanced developments and 
state-of-the-art innovations throughout the aerospace industry. 

Data Inputs 

The preliminary system-level review requires, as data inputs, all existing documentation of 
system design limitations. This includes such specification items as: 

(1) System performance 

(2) Environmental criteria 
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(3) Interface definitions 

(4) Acceptance criteria 

(5) Vendor control definition 

(6) Top drawings 

(7) Test specifications 

(8) Reliability predictions 

(9) Failure mode and effects analyses 

(10) AGE (aerospace ground equipment) and facilities requirements 

(11) Design criteria reports 

Reliability program plans (NPC 250-1, par. 2.2.1, ref. 2), quality assurance plans (NPC 200-2, 
par. 3. 1. 1, ref. 3), and similar items are other necessary inputs supplied by the applicable 
project management group. Since these documents will be in various states of completion and 
approval, the contractor proposal may be needed in some areas to supplement the information. 
Trade-off study reports from all design groups which illustrate design concepts discarded in favor 
of the present configuration (e.g. , feasibility studies) are major data inputs. 

In larger programs where significant components will be procured, it is usually necessary to 
have a "vendor control specification" to be contractually applied to subcontractors. In addition to 
the contractual basics, this document provides general technical information to enable compliance 
with reliability, quality, and documentation requirements. Provisions of this document should be 
reviewed for adequacy. 

In the case of a second or subsequent item (spacecraft or launch vehicle) in a series, where 
there is significant design modification from item to item, the input documentation at the prelim- 
inary design review should include a summary of failure experience and correction in the late 
stages of testing and field use of the previous item. 

Data Outputs 

The output from a preliminary system -level design review is a comprehensive report docu- 
menting the conclusions reached during the review. Conclusions take the form of recommenda- 
tions to the program management that specific portions of the system engineering effort be ap- 
proved or that certain system concepts are unsubstantiated or are based on unsound analysis and 
need either substantiation or revision. As backup for the conclusions, the review report will: 

(1) List all documents forming a portion of the system criteria by which specific subsystem 
requirements have been defined and from which subsystem design evolution will proceed 
(See preceding section on data inputs for specific documents) 

(2) Assert that the boundaries defining system performance are completely and accurately 
established 

(3) Document the logic by which other system concepts were analyzed and then discarded in 
favor of the selected approach and the trade-off considerations which enter into the selec- 
tion of this approach 

(4) Present alternatives to the recommended courses of action and their relative effective- 
ness (or ineffectiveness) in achieving mission objectives 

When selection of the desired system is not fully substantiated, the recommendation may 
favor parallel development until the correct choice is clearly indicated. As a practical matter, 
the project manager may elect to make a selection among the alternate systems on the basis of 
some criteria other than technical. It is unlikely that this decision could be made without further 
thought, customer negotiation, or research, since the review team presumably had all current 
information available when the concurrent decision was reached. 
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Subsystem Level 


Purpose and Scope 

A preliminary design review is performed for each subsystem when its definition has pro- 
gressed to the point that functional diagrams, schematics, block diagrams, and major component 
functional diagrams (or requirements sheets) are complete. The review assures that the pro- 
posed subsystem does satisfy the system requirements and objectives and that the schematics 
and proposed component requirements represent a technologically sound solution. The review 
may be a part of, or extension of, the system preliminary design review. 

The subsystem preliminary review will prove most effective if it can be performed prior to 
initiation of negotiations for component subcontractors. (Basic system engineering precepts re- 
quire complete definition of requirements independent of influences from existing hardware capa- 
bilities.) The exception to this rule occurs when a particular subcontractor has been included in 
the definition phase proposal. Where such a contract obligation exists, the characteristics of 
that component are a program constraint and can only be altered by customer direction. If re- 
view determines that basic program objectives are incompatible with continued use of the com- 
ponent, the recommendation to contractor management should request a formal detailed presen- 
tation to the customer for a contractual change. 

It is recognized that subsystem development is an iterative process and that inputs from the 
component subcontractor must be fed back into the subsystem design; also, the desirability of 
use of an existing proven component may dictate changes to the subsystem concept. Occasional- 
ly, feasibility of a component must be established before a meaningful subsystem review can be 
held, and review schedules should be rearranged accordingly. 

Preliminary design review at the subsystem level would not be complete in most cases with- 
out a review of the AGE subsystem. The major considerations of this review are the compatibil- 
ity of the test and maintenance facilities with the flight hardware and their effect on operational 
readiness, safety, and performance. 

This review should also include all significant test and maintenance facilities required to 
provide assurance data. These requirements become more and more complex as the gravity of 
the mission (i. e. , the "program significance") increases. Although the evolution of support 
equipment lags behind that of the operational system (since the requirements for the former are 
determined from specific parameters of the latter), it is still necessary that the feasibility of 
the concept for such items as the checkout equipment be evaluated at least grossly at the same 
time the subsystem is evaluated. An important consideration of the review of test and mainte- 
nance facilities is the compatibility of interface parameters to avoid compromising operational 
system reliability. It will usually be the case that the checkout and maintenance equipment does 
not receive the design and management attention that the flight equipment does. However, it is 
essential that checkout equipment which must check the failure modes most probable in the flight 
equipment must not have, in itself, failure modes which have a high probability of accepting 
faulty flight equipment or of damaging good flight equipment. 

Review Team 

The subsystem preliminary design review requires much the same depth of personnel exper- 
ience as does the system review; however, the aggregate knowledge centers on the technologies 
pertinent to the subsystem under review. In smaller companies the chief engineer will probably 
act as chairman; in larger companies the chairman will be selected from the ranking personnel 
in the particular technologies, such as heads of technical departments. Project personnel in a 
direct line of authority to the designer should not serve as chairmen, since they would be review- 
ing themselves. At this level, the basic contractor team may be self-sufficient but should not 
ignore the possibility of consultant inputs. Systems engineers who participated in the system 
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review can contribute a degree of continuity to this review and thus reduce the necessary prepara- 
tory effort. The systems engineer will be alert to the possibility of omissions or incompatibili- 
ties in the subsystem definitions and constraints thus far established and to iterative requirements 
for modification of system definition. 

Data Inputs 

Data inputs include functional flow diagrams, task breakdowns, test specifications, design 
specifications, design criteria reports, schematics, and component requirement sheets for the 
subsystem in question: 

(1) The functional flow diagrams will present subsystem definition in terms of those func- 
tions and their interfaces required to attain one or more mission objectives. Customer 
interface specifications should be integrated into the functional flow diagrams where 
another contractor's subsystem adjoins (physically or functionally) the subsystem under 
review. 

(2) Task breakdowns of each function will be available from the initial design efforts. The 
breakdowns should show continuity and completeness of coverage through to the proposed 
subsystem solution as portrayed on schematics and component requirement sheets. 

(3) Status summaries of parallel development efforts are required, along with customer doc- 
umentation of any revisions to mission objectives. 

(4) Updated contractual specifications should be available for reference to conditions appli- 
cable to the reviewed subsystem. 

(5) The preliminary structures review requires inputs as to availability and adequacy of 
manufacturing facilities and plans for test of vehicle structural models. 

(6) Reliability prediction, apportionment, and failure mode and effects analyses are 
important inputs. 

(7) If there have been advance feasibility studies of a critical component, a report of those 
findings is necessary to reach any conclusions as to acceptability of the subsystem con- 
cepts which include that component. 

(8) General reference documents include planning reports for reliability safety, logistic 
support, quality assurance, manufacturing, test, facilities, human engineering, etc. 

The policies in these plans were established as program measures to assure that mis- 
sion objectives would be met and should carry over into the subsystem definition 
documents. 

Data Outputs 

The report of a preliminary subsystem review should furnish a level of confidence in the es- 
tablished functions and design criteria of the subsystem and its major components and in the 
schematic representing the proposed design solution. 

Report recommendations should cover every phase of the review discussions and should cul- 
minate in suggested approval or disapproval of the specific documents comprising the subsystem 
design. Where individual specialties (reliability, maintainability, human engineering, safety, 
etc. ) propose design revisions, a complete rundown of the advantages and disadvantages of alter- 
nate trade-off possibilities is advisable to facilitate the management decision. 


Component Level 


Purpose ana Scope 

All C omp orients . The preliminary review at the component level consists of a review of re- 
quirements which will be expressed at this point with some formality as a preliminary component 
specification, i.e. , a series of demonstrable criteria. The review is intended to assure that the 
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criteria expressed in the component specification do satisfy the subsystem and system 
requirements. 

In addition to the obvious physical and functional parameters of the specification, the review 
must cover the subtleties of performance, interface, testing, and handling which can grossly 
affect the reliability and cost if not adequately specified. These subtleties might include such 
items as duty cycles, susceptibility to or generation of electromagnetic interference, validity of 
qualification test environment and sample size, adequacy of test points and test equipments, 
validity of acceptance techniques, storage and shipping provisions, and life or cycle limits. 

For the more complex components, the subtleties should also cover the determination of 
testing and data requirements necessary to verify the adequacy of part and material applications; 
however, because of its detailed nature, this determination can best be made in a separate pre- 
liminary parts and materials application review, the results of which are presented as an input 
to the component design review. 

Qualitative criteria in a component specification should be avoided, and quantitative criteria 
should be demonstrable. When a reliability number is specified (i.e., required), means of dem- 
onstrating compliance must be included. This might be analysis, limited no-fail testing, or 
testing to prescribed confidence limits. Confidence should only be specified in connection with a 
specific demonstration. For components produced in-house, the result of the review should be a 
confirmation or refinement of the preliminary component specification. 

Components To Be Subcontracted. When the component is to be subcontracted, preliminary 
component design review evaluates the initial specification for compliance with subsystem and 
system requirements, and formal vendor proposals, vendor selection, and subsystem contract 
negotiations develop the ultimately released component specification. For items in this category 
it is sometimes prudent to have the initial contacts with potential suppliers prior to the prelim- 
inary review, particularly if the contractor is not well versed in the particular technology. If the 
potential vendors uniformly indicate strenuous objections to a specific criterion, it is necessary 
to review the objectionable requirement at subsystem or system level for possible relaxation or 
system change. (In such a situation, an isolated vendor who will promise compliance should be 
regarded with considerable caution; one must be sure that his promise is soundly based and will 
not result in a problem later on.) 

It is not intended that the preliminary design review replace, or even enter into, the normal 
selection process for component vendors. At review time, the component specification has not 
yet reached a negotiable form and is sometimes given a separate designation for procurement 
purposes. 

Although vendor inputs and negotiations are useful in establishing realism in the component 
specification, they must not be made governing factors in relaxing component requirements es- 
sential for meeting important requirements of the subsystem or system. 

The subcontract specification must clearly define all items necessary for compliance with the 
provisions of the system reliability, quality, and maintainability program plans. Specific refer- 
ence (objectives and data requirements) toprepackaging, prerelease, postqualification, and ac- 
ceptance reviews will ensure vendor awareness of this design tool during the negotiations. This 
will facilitate his program planning and understanding of contractor needs and also will permit 
early discussion and resolution of proprietary arguments. 

Review Team 

Makeup of the review team for a preliminary component review varies with the complexity 
and significance of the component to the subsystem design. A major component essential to 
safety and mission success warrants the attention of divisional staff specialists. 

Customer attendance follows the same line of reasoning. Where complexity and program 
significance are great, the customer’s industry-wide background in many fields can be utilized 
more fully. However, his presence in any preliminary review should be with adequate 
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preparation and applicable experience. Availability of the right customer representative for a 
particular component review might pose a practical limitation on customer attendance. 

Data Inputs 

The primary inputs to the component preliminary review are: 

(1) Component specification 

(2) Subsystem design criteria report 

(3) Preliminary vendor inputs 

(4) Studies and analyses completed since subsystem review 

(5) Parts and materials application review report (for complex components) 

The subsystem preliminary review report, with disposition of decision items pertinent to this 
component, will provide a reference point for initiating the current review. Much modification 
and refinement of the component requirements through continued design efforts, subcontractor 
contacts, etc. have most likely occurred since the subsystem review, thereby dictating a re- 
evaluation of component concepts against updated mission objectives. Customer directives and 
documentation of contractor action should be evaluated during the review preparatory effort for 
effect on the component or its interfaces. Reference material that is sometimes required in- 
cludes contractual specifications and plans for applying principles such as safety, maintainability, 
and reliability to system design. 

Data Outputs 

Satisfactory completion of a preliminary component review implies that the component defini- 
tion as evolved is compatible with the system and subsystem requirements and with the physical 
limitations imposed by the design. The review report presents a critique of the component speci- 
fication and an appraisal of it in relation to the requirements outlined in the foregoing section 
entitled "Purpose and Scope.” Where alternate approaches to component design have been selected 
by the review team as being clearly superior to existing concepts, the report must provide pro- 
gram management with a description of the available alternatives, including trade-offs, design 
problems to be solved, program impact, and so forth. Finally, the report should document the 
exploration of questionable areas which did not result in change recommendations (since present 
design was found to be justified) . 

A management decision at this checkpoint is necessary only if review findings indicate a 
change is necessary. Upon concurrence with these findings, management will direct appropriate 
action by the pertinent design activity. In cases of nonconcurrence, a written explanation must be 
appended to the review report. 

PREPACKAGING AND PRERELEASE DESIGN REVIEWS 

Prepackaging and prerelease design reviews performed by the contractor will normally be 
limited to component levels. Although these reviews can be, and sometimes are, conducted at 
subsystem and system level, system reviews at this time in most programs tend to become un- 
wieldy and do not provide the detailed look that is required. When it is performed at a higher 
level, it is usual to break the review into a number of smaller segments to be performed concur- 
rently by the cognizant customer and contractor personnel. 

Prepackaging Review 


Purpose and Scope 

A prepackaging review is held whenever a clearly defined design milestone exists between 
preliminary and prerelease reviews . The prepackaging review normally is limited to electronic or 
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electromechanical components, where significant design decisions of circuit selection and analy- 
sis, part selection and application, and breadboard testing are clearly and necessarily separated 
from (and completed prior to) the packaging design effort. This review is also appropriate for 
science experiments on spacecraft programs. The review is intended to assure that the com- 
pleted circuit will perform the required functions, will remain stable under all operating condi- 
tions, and has been balanced with respect to trade-offs of simplicity, standardization (circuit 
principles and parts) , cost, reliability, power requirements, tolerances, and so forth. The re- 
view will assure that the application of parts and materials has been adequately reviewed. (See 
NPC 250-1, par. 3.9.6, ref. 2; this ancillary review is also described in an as yet unpublished 
document of this series.) 

Packaging concepts should be analyzed as thoroughly as possible during the prepackaging re- 
view to insure consideration of such fundamentals as radio frequency interference control, heat 
dissipation, mounting requirements (shockmounts, alinement, etc.), weight, geometric space, 
pressurization, fuel compatibility, and so forth. It is apparent that the best parts, even though 
correctly applied from an electrical circuit analysis standpoint, still require proper packaging 
to operate under the environmental conditions on which the reliability predictions are based. 

Review Team 

The prepackaging review requires a small, highly specialized team, consisting of electrical 
or electromechanical design specialists (circuits and packaging) and a reliability engineer. The 
design specialist must specifically not be the designer of the component under review but an en- 
gineer experienced with circuits of that particular functional category. Hardware -oriented per- 
sonnel are best suited to this effort. Across-the-board customer participation may be difficult to 
achieve. However, customer representative attendance should be stressed in specific reviews of 
high -complexity components. 

Data Inputs 

The major inputs to the prepackaging design review are: 

(1) The electrical circuit schematic, component specification, and packaging concepts (from 
the design group) 

(2) Parts and materials application review reports 

(3) Reliability analysis and the applicable design criteria report (from systems engineering) 

(4) Other supporting data furnished by the designer including circuit analyses, breadboard 
test results, and pertinent design notes 

Data Outputs 

The report of a prepackaging review will do one of the following: 

(1) Recommend proceeding with the component packaging detail effort on the basis of accept- 
able circuit design 

(2) Include the logic of rejection of the present approach and suggested alternatives 

It will also include data substantiating the positions taken. Acceptability of the parts and mate- 
rials application review is recorded along with conclusions regarding completeness and accuracy 
of analysis and test data. Rejection arguments should establish the impact of present design on 
subsystem operation, so that the potential problem may be defined. Precautions in packaging 
should be noted in order to provide inputs to both the packaging designers and subsequent design 
reviews . 

The report is submitted to project management, specifically, the assistant technical director 
responsible for design drawing signoffs, for action as indicated above. 
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Prerelease Review 


Purpose and Scope 

Prerelease reviews are held just prior to the release of engineering drawings for manufac- 
turing. They are applicable to all elements of the system including science experiments. They 
provide the greatest potential for discovery of detail problem areas. Here, as with prepackaging 
reviews, the activity usually is conducted at the component level. At this time the designers con- 
sider their design to be complete; all development and evaluation tests have been completed. The 
output data from the prior reviews, including action items, are available. Only qualification 
testing to demonstrate that the design has its specified capability remains. This prerelease re- 
view is the last chance to prevent premature submission of an immature design to qualification 
testing. (Historically, the designer’s confidence is seldom justified and changes will be required 
as a result of qualification testing. During the Gemini Launch Vehicle qualification program, for 
instance, components experienced 176 failures in 962 tests, and the Mariner MM-64 experienced 
58 failures in 805 tests (ref. 4).) 

The prerelease review will be directed to the detail hardware and will cover the following 
points: 

(1) Has the packaging altered the circuit characteristics (previously reviewed in the pre- 
packaging review) ? 

(2) Has the designer considered the qualification test as a design requirement (and possibly 
the most severe requirement) ? 

(3) Have the parts and materials application data been updated to include latest configuration 
and part-use data? 

(4) Did the evaluation testing really evaluate the hardware relative to its capability for 
passing qualification? 

(5) Where qualification by similarity is claimed, are both the hardware and the usage really 
similar to the cited example? (An item may have been previously qualified but may now 
require additional testing because of changes in mission environments.) 

(6) Can this design be manufactured, inspected, and readily tested? 

Included are reviews of specifications for manufacturing checkpoints, acceptance test environ- 
ment, quality controls, and qualification test stresses, as well as the storage, installation, trans- 
portation, ground test, and flight environments. Results of prototype manufacture and test are 
necessary inputs to this review in order to obtain a preview of the probability of success of the 
manufactured version. These questions are not, of course, intended as a check list but only to 
indicate the direction this particular review should take. 

The similarity decision, touched on above, is important to many NASA programs. The argu- 
ments for use of off-the-shelf components are economically luring and often technically valid; 
however, we seldom get them truly off-the-shelf. Lapses in production, incorporation of seem- 
ingly innocuous changes, or differences of usage can adversely affect the performance history we 
think we are buying. Review must be carefully performed. 

The limited production quantities frequently required by NASA contracts induce additional 
problems. The unique tasks frequently require nonstandard components. The development of 
these components never achieves full maturity, and it is not economically feasible to design for 
mass production. Random selection of the qualification units from the production lot is not feasi- 
ble since reason demands that qualification be complete prior to the acceptance review. The first 
units off the line are usually allotted to qualification testing. Sometimes schedule and budget 
commitments dictate the use of a prototype unit for testing if it has been built by the manufactur- 
ing department rather than a model shop. A certain amount of hand grooming during manufacture 
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is not necessarily bad, unless the qualification units begin to receive some extra touches not 
afforded the delivered hardware. Frequently, qualification is achieved but subsequent "builds” 
cannot pass acceptance testing. 

These problems should be anticipated and the design reviewed from a producibility standpoint 
in the prerelease review. Every effort should be made to relax manufacturing tolerances and 
eliminate difficult process requirements if other trade-off considerations permit. The time for 
manufacturing inputs to the detail design is prior to drawing release, and design review should 
assure that this critical function is performed. 

The prerelease review also exercises the tact and diplomacy of the review team to a maxi- 
mum. The designer feels he has finished, and always has an imminent schedule release date. 

He is ready to go to the next job and does not wish to answer a lot of "unnecessary” questions 
about the one he just finished. However, as indicated above, this review is extremely important 
and must be carried to a conclusion. 

The prerelease review is applicable to all major components of the flight operational system 
and any items of support equipment considered critical to launch safety or mission reliability. It 
is frequently not conducted at subsystem levels since release of the detail component drawings 
within the subsystem may occur at widely different times. 

Review Team 

A team of hardware -oriented personnel, including some members of the prepackaging review 
team, is needed to accomplish the prerelease review. In addition to the designer, a hardware 
design specialist, a manufacturing tool designer, a reliability engineer, and an environmental 
test engineer provide the necessary skills for most of the required evaluation. Additional con- 
tractor .participation may be required and secured during preparatory activities for specific re- 
views; this may include written inputs or direct participation by maintainability, human factors, 
safety, materials, quality control, value, and logistics personnel. Here, as with prepackaging 
reviews, customer attendance across-the-board may be difficult to achieve because of the large 
number of components involved. 

The treatment of a vendor -built item must be handled discretely. A subcontractor team will 
frequently visit the contractor plant to present the design. This team probably will include senior 
technical representatives (particularly from smaller vendor houses) and a sales representative to 
protect the vendors proprietary data. (In some cases, this complication requires that the review 
be conducted at the vendor facility.) Access to vendor internal drawings for design review pur- 
poses should be negotiated and assured at the time the work is contracted to facilitate the 
scheduled reviews. 

Data Inputs 

The primary input to the prerelease review is the completed set of assembly drawings show- 
ing the planned package configuration of the production component. Review attention centers on 
this compilation of detail information to evaluate the completeness and effectiveness of the pro- 
posed design solution with respect to the component test specification and to system performance 
requirements. The remaining data inputs provide support information on which to base a decision 
to accept or modify the assembly (detail) drawings. Included in the data package will be: 

(1) Component procurement specifications 

(2) Prototype test results 

(3) Documentation of previous reviews 

(4) Parts and materials application review report 

(5) Reliability predictions 

(6) Failure mode and effects analyses 

(7) Manufacturing processes and techniques 
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(8) Cost analyses 

(9) Qualification schedules and procedures 

(10) Alternate designs 

It is suggested that actual models be made available to team members for visual evaluation, 
if at all possible. A prototype test unit, partially disassembled, will assist in focusing review 
team attention on specific design areas. Even if the model is only a preliminary version, it will 
provide a basis for visualizing the final drawing configuration. Prior use of similar components 
should not be overlooked as a possible source of representative hardware. 

Data Outputs 

The end product of the prerelease review is a recommendation to contractor management 
that the component detail drawings be released for qualification testing or that specific design 
revisions are needed prior to release. The recommendation is substantiated by a point-by -point 
evaluation of the effectiveness of the component. Affirmative answers to the general questions 
posed in ’’Purpose and Scope” at the beginning of this section are essential, so each detail decision 
must reflect the component capability of being produced, performing the required functions, and 
surviving qualification environments. 

Although the usual function of the review output documentation is to provide advice to the 
project technical management and to assure retention of the review proceedings, the component 
reviews in particular can provide cogent inputs to the reviews of similar components. For ex- 
ample, one group engineer avoided a diode problem that plagued all the other groups designing 
similar equipment. His novel treatment was completely missed during the design phase and was 
not recognized until analysis of failures on the other components was initiated. Adequate design 
review output records could have informed those concerned much sooner. 

ACCEPTANCE REVIEW 
Purpose and Scope 

The acceptance review is the primary assurance to the customer that he is receiving the 
product he specified. Although the term ’’customer acceptance" is usually considered synony- 
mous with "Government acceptance," the principles of this review should also apply to prime 
contractor acceptance of major subcontractor articles. The acceptance review is technically 
equivalent to the First Article Configuration Inspection (FAC I) which is normally associated with 
a mass production contract. However, the usual characteristic of a NASA contract is limited, 
frequently singular, production. When more than one article is involved, the individual articles 
usually vary in configuration and flight objectives, so that an acceptance review and buy-off is 
necessary for each article. 

Since there is available a NASA management guide, NPC 500-1 (ref. 5), which develops the 
FACI review for use on Apollo, we will start with that definition. Users of the present publica- 
tion can modify that definiton to suit second and succeeding production articles; this will depend 
on the amount of variation from the first article. 

FACI is defined by NASA NPC 500-1, "Apollo Configuration Management Manual" (ref. 5, 
exhibit XIV) , as "a formal technical review which establishes the product configuration baseline." 
This reference goes on to state the specific tasks of FACI to be: 

(1) Determine compatibility between released engineering and as -manufactured configuration 

(2) Determine compatibility of qualified configuration to (1) above 

(3) Determine validity of acceptance testing by comparison of test method and test data with 
the unit’s performance and design requirements 

(4) Insure that the initial submittal of customer-requested drawings and records be either the 
same as (or revisions of) those submitted for audit at the FACI 
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(5) If circumstances warrant, review and validate engineering release system and change 
control procedures 

The FACI, therefore, consists of a step-by-step demonstration by the contractor that he is 
in compliance with the procurement specification s) . Because of the customer buy-off function as 
the result of this review, the output must be a simple "Yes, it does comply" or "No, it does not 
comply." Resolution of the latter result must be included in the review report. This might be in 
the form of corrective action to be accomplished or, in some cases, waiver items on DD Form 
250, entitled "Material Inspection and Receiving Report." 

The review may be divided into segments where the interactions can be separately evaluated. 
However, the segments must be clearly defined with no contingencies or exclusions. Prepara- 
tory effort requires reviews of individual components, since the documentation and data assembly 
task is too large collectively to be performed by a single acceptance review. It should also 
include physical inspection of the hardware to gain assurance of the quality of mechanical design 
and workmanship and to resolve mechanical questions which are best seen in three dimensions. 
Outputs from these preliminary reviews state "Yes, this component complies" or "No, these 
tasks must be completed prior to the formal review;" these statements are followed by a listing 
of the review findings. Then, too, the formal review may be facilitated by separation into sub- 
systems and by individual teams pooling their findings at the conclusion of the separate reviews. 
The manner of segmentation is arbitrary as long as the summed results prove complete 
compatibility. 

The scope of the formal acceptance review depends to a great extent on the thoroughness of 
documentation of the contractor’s prior design review program. With an adequately documented 
review program, the formal acceptance review might cover only the system level and acceptance 
testing. 

Review Team 

The acceptance review is performed by top engineering and management personnel of the 
cognizant NASA installation, defined by NASA NPC 250-1 (ref. 2) as "that major organizational 
unit of NASA which has direct technical and managerial responsibility for the system under con- 
tract." This may be either a project at a NASA center or a program office at NASA Headquar- 
ters. Prior to review by either agency, there are many contractor internal reviews for the 
preliminary collection and organization of acceptance data. These meetings are usually conducted 
by contractor test integration personnel since the bulk of data gathered here is test-oriented. In 
this type of review, the designer is assisted by personnel from reliability, test, quality, and 
manufacturing to present supporting information as to hardware compatibility. The efforts cul- 
minate in a preparatory review by the contractor program manager who, in turn, leads the pre- 
sentation to the customer acceptance review team. 

Data Inputs 

In order to determine compatibility between released engineering and as -manufactured con- 
figuration, data inputs must include updated assembly drawings, detail drawings, test specifica- 
tions, manufacturing processes, and quality approval sheets for all manufacturing and assembly 
steps. Previous reviews may have produced adequate documentation to show compatibility be- 
tween manufacturing procedures and released engineering as well as the capability of that 
engineering design to satisfy mission objectives. Internal contractor reviews now perform the 
step-by-step compliance inspection of the quality records of the manufactured article. Documen- 
tation of these reviews becomes part of the data package for the customer acceptance review. 

For determination of the compatibility of the qualified component configuration with the manu- 
factured version in the first article, the quality records for qualification test units are also sub- 
mitted to the internal review. Any manufacturing changes resulting from qualification test failures 
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are verified as having been included in the manufacturing procedures for deliverable hardware. 
These review findings as well as qualification test reports and customer signoff sheets also be- 
come part of the customer acceptance data packages. 

All associated failure reports are presented in a problem status summary to indicate closure 
or implemented corrective action leading to closure of problems affecting the manufactured arti- 
cle. Individual failure reports are available on request of the review team. Reliability predic- 
tions and failure mode and effects analyses have been updated to include the contractor's experi- 
ence during the development cycle. When the item is second or later in a series, failure 
experience and correction occurring too late to be covered in the flight readiness review of the 
preceding item should be another input at this point. 

The system's test log, including final acceptance test in the presence of the customer, com- 
prises another major input to the formal review. As previously stated, the validity of acceptance 
testing is determined by comparison of the method and data obtained with the performance and 
design requirements. 

Configuration control procedures are sometimes submitted at this time as proof that subse- 
quent system changes will be adequately documented and incorporated into existing hardware. 

Data Outputs 

The most significant data output from the acceptance review is the "Material Inspection and 
Receiving Report," DD Form 250, MSFC Form 71 (ref. 6) or an equivalent, constituting customer 
acceptance of the manufactured system. This document defines the terms of acceptance, includ- 
ing successfully completed requirements and waiver items. Waiver items describe conditional 
acceptance of the system although deviations from specified criteria exist. The waiver permits 
shipment and additional testing to occur concurrently with corrective action for the deviation. All 
waivers must be cleared prior to flight test (use) of the system. 

The data compiled for an acceptance review might be considered to be an output in itself. 

The assembled qualification test and configuration data can be formalized and maintained in an up- 
dated condition to show current component configuration and its traceability to the qualified unit. 
Use history can be tabulated (if it has not been done previously) and appended by further usage. 

In short, the acceptance file can be updated as changes occur in order to facilitate flight readi- 
ness reviews or acceptance of future systems. It is possible that, if significant vehicle configur- 
ation change occurs, additional FACI's may be required. 

SPECIAL PURPOSE REVIEWS 

Several categories of review which might be considered outside of the scope of design reviews 
have become recognized techniques and are described briefly. 

Flight Readiness Review 

Immediately prior to the launch of any vehicle or payload, a final review is held to assure 
customer management that the system inherent reliability has not been degraded because of any 
laxness in technical or documentary discipline and that test experience subsequent to acceptance 
review has created no significant doubt of the capability of the flight hardware successfully to 
perform the mission. The review (or series of reviews, depending on the complexity of the sys- 
tem) is performed by teams of high-level personnel with sufficient authority to enforce their de- 
cisions. The team members are frequently, and most effectively, not directly connected with the 
program. 

In programs of major complexity, it is usual for the prime contractor or the integration con- 
tractor and the major associates to have a review in depth prior to the customer's review. In the 
Gemini Launch Vehicle Program, for instance, the Launch Integrity Team (LIT) composed of 
high-level representatives from the several associate contractors reviewed each launch vehicle 
in depth prior to the flight safety review meeting with the customer. A similar technique was 
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used by the contractor for the Atlas -Mercury launch vehicle. The flight readiness review pro- 
gram for the Apollo spacecraft program is defined in ref. 7. 

The major input for this review is the problem status summary. Since the review team can- 
not cover a system to any real depth, it must establish its confidence in the system readiness by 
evaluating the program personnel’s understanding of recognized problems, evaluating the efficacy 
of the fixes, and probing for soft areas in the technical story or its documentation. Obviously, 
documentation of the close-out of any waivers listed on the prior acceptance review (DD 250) must 
be presented. 

Output is normally formalized in a relatively brief report since it will not be required as an 
input to other reviews. The output decision is whether or not to launch. It is obvious that the 
team needs considerable authority and stature for a "no-launch" decision. In the case of the con- 
tractor in-house reviews such as the LIT, the output is, in addition to the written report, a 
series of oral presentations to the customer review team. 

System Readiness Review 

The readiness review is a customer management function intended to provide a basis for buy- 
off of a particular portion of system assurance testing. The buy-off implies an integrity to 
proceed with further assurance functions. It is based generally on verification of the hardware and 
the test facilities and procedures to be used in the next event; often, it also covers certain critical 
safety aspects of the design. Specific detailed objectives include assuring that (1) no quality 
discrepancies are unresolved, (2) all prerequisite activities are accomplished, (3) no failures 
remain unexplained or uncorrected, and (4) a thorough set of controlling documents and proce- 
dures exists for the next assurance step. Particular check points include consent to test (CTT) 
and consent to ship (CTS). The acceptance review described in a previous section under 
"Acceptance Review” is the functional equivalent of CTS; the flight readiness review (above sec- 
tion) is a special case of CTT. 


Postqualification Review 

Design review is conducted upon completion of component qualification and prior to submittal 
of the qualification report to the customer to provide assurance that design and test revisions 
which have been made since release have not adversely affected design effectiveness, that all 
testing to date confirms the adequacy of the design solution, and that all changes to qualification 
units are reflected in the production hardware. 

The review considers the results of qualification and special testing, the problems of proto- 
type production, and all change control board action on the component detail drawings and test 
specifications since the last review. The value of this review increases with the amount of diffi- 
culty observed in qualification. The pressures to redesign and retest after failure can become 
quite intense and many things are accomplished with "paper work to follow." If further failures 
occur, the paper-work lag is increased. Additional complications are introduced by time lag and 
communications problems among the contractor, vendor, and test agency. At the completion of 
qualification testing the required number of units might have passed, but the lack of proper docu- 
mentary control has resulted in extremely obscure development history. The task of design re- 
view ensures that (1) changes to the released configuration were qualified and were made prior to 
full production, (2) changes were inconsequential, or (3) retrofit of affected units has been under- 
taken. The postqualification review is conducted on each development component in the system. 

The postqualification review team utilizes two types of personnel: 

(1) Engineers experienced in environmental test and evaluation— those able to determine the 
cause and significance of test failures, fixes, retest validity, deviation from standard 
procedures, and so forth 
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(2) Engineers able to evaluate the effects of test results on subsystem performance (possibly 
members of the prerelease team) 

The design is represented in this review by the qualification test engineer as well as by the 
designer, since both parties have first-hand knowledge and experience in testing the component. 
The manufacturing planner responsible for production hardware provides configuration informa- 
tion. The review team nucleus, as previously indicated, might consist of senior design, reliabil- 
ity, quality, and test engineers, preferably (but not necessarily) dissociated from the project. 

In view of the contractual significance of the submittal of the formal qualification test report, 
customer participation in this review is not appropriate. The review is basically an internal 
work meeting designed to ensure the technical adequacy of qualification data. Customer attend- 
ance would tend to alter the meeting objective to that of an initial sales presentation. In this 
atmosphere, it would be impossible to evaluate objectively technical flaws which might require 
requalification, should such be the findings of the review team. The design review report will, 
of course, be available for customer review at his request roughly 1 week after the review. At 
this time, he can coordinate any comments or suggestions as to the data presentation in the for- 
mal qualification test report with the contractor test group. 

It might be pointed out that customer signoff of the formal qualification test report approves 
the test of that particular component but does not imply concurrence with the system configuration 
which utilizes that component. The buy-off of system configuration occurs following the accept- 
ance review. 

The primary inputs to this review are the qualification test specification and procedure, data 
sheets and notes including any existing failure reports, failure analysis reports, and problem 
closures which are to be formalized into the qualification test report. Also included in this 
package are component drawings, procedures, and quality logs utilized in the manufacture and 
acceptance test of the qualification units (and deliverable hardware, if production is already in 
progress). Special production and test data (e.g. , prototype units) not previously covered should 
be reviewed. Prerelease design review recommendations and management followup action will 
provide a reference for present discussions. Associated change control board paper work should 
be available. The tested hardware should be made available to the review team to permit on-the- 
spot evaluation of damage and rework. 

As with previous reviews , the output will be a formal report to contractor management, gen- 
erated by the review chairman, of the findings and recommendations with regard to input data. 

In this particular review, the major outputs deal with qualification and production phases of com- 
ponent acquisition. 

The major output involving qualification could range from unconditional acceptance of the 
basis, procedure, and results to recommendation for partial or complete requalification. A sec- 
ond, equally important output finds complete conformance of production configuration to that 
qualified or recommends specific modifications to production procedures (and hardware if pro- 
duction is already in progress) . 

In addition to the immediate assurance benefits of the postqualification review, the output 
serves as part of the preparatory effort for the acceptance review. Documentation of this review, 
appended by the qualification test report, customer signoff sheets, and later similarity supple- 
ments for minor design revisions, is the basic input to the impending configuration inspection by 
the customer. 


Subsystem Status Review 

Purpose 

The subsystem status review series establishes an assurance that all system study results, 
altered system objectives, changes to component design, development test data, and 
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state-of-the-art advancements are receiving proper evaluation for pertinency to the subsystem 
configuration and that the configuration has been revised when necessary but not degraded in the 
process. 

The occasion for reviews of this type sometimes arises in the development cycle of major 
systems when the time between previously identified milestones is quite long. Here, in view of 
the continuous influx of new information pertaining to system development at all levels, design 
review by other-than-project personnel is advisable on a periodic basis. For practical purposes, 
it is best to conduct these reviews by subsystem; component reviews are too numerous and sys- 
tem reviews, too unwieldy. The frequency of these status reviews would vary with the state of 
flux of design in each subsystem; however, a suggested interval would be every 6 to 8 months 
after the subsystem preliminary review. 

Review Team 

Technical department staffs of the contractor organization provide the personnel to conduct 
these reviews; preferably, they should be the same as those specialists involved in the preliminary 
subsystem review. The talents required will conform to the type of data predominantly available 
for review, progressing from theoretical studies to actual test results as development proceeds. 
The design is presented by the project engineer assigned responsibility for the subsystem under 
review. 

The series of reviews affords ample opportunity for customer technical representatives to 
keep pace with contractor efforts; however, adequate preparation is again emphasized as an es- 
sential prerequisite to participation. 

Input Documents 

Inputs for this review are the subsystem specification, schematic, and description report, 
each updated to show current status of development. Specifications of major components are 
frequently applicable. 

Supporting data cover a wide range of document types and actually include all study and test 
reports resulting from the design and development effort as well as a performance history of re- 
lated (similar) projects. Problem investigation reports receive special attention elsewhere in the 
program organization and so are not stressed herein. Changes evolving from problem investiga- 
tions will be reflected in the configuration data, and investigations in progress are best deferred 
to the next status review. 

Assurance of complete data availability will be obtained in a manner consistent with the con- 
tractor organization. In general, PERT summaries or engineering directives will permit 
traceability of study and test data. Followup on prior review recommendations as acted upon by 
program management will provide a reference for initiating current discussions. 

Output Report 

The report of a subsystem status review will not follow a specific pattern since the type of 
data reviewed will vary during the course of development. The report will, of course, list data 
inputs and personnel, but items of its content will depend on specific areas of inquiry opened by 
the reviewers. The end product will be a judgment of design adequacy with possible recommen- 
dations of retest or additional test or study efforts in areas considered technically weak. 

Functional Review 

Occasionally, after the system preliminary review, situations will arise requiring reevalua- 
tion of some facet(s) of system configuration from a functional viewpoint rather than by physical 
subdivisions. The purpose might be to evaluate the results of special studies or tests or to judge 
the method of implementation of revisions to program philosophy. The material covered by this 
type of review usually encompasses more than one subsystem. A special function review would 


REVIEW CATEGORIES 


31 


be held, for example, to reevaluate prelaunch hold and shutdown parameters, to consider weight 
reduction techniques, or to update reports on malfunction hazards, flight performance improve- 
ments, the master measurements list, and so forth. Such reviews would be conducted on an as- 
required basis by nonproject personnel whose capabilities would be defined by the function(s) being 
reviewed. 


Failure Review 

Reviews following failures are not subject to any specific set of directions. Extreme disci- 
pline is required to prevent the review from losing its objectivity because emotions tend to run 
high and potential contributors are still in a state of shock. Initial action requires the selection 
of a chairman with recognized authority who will impose this discipline on customer and contrac- 
tor personnel alike. 



CHAPTER 4 


Application 

The design review program for any particular contract is, of course, a subject of discussion 
and agreement between the contracting agency and the contractor. NPC 250-1, par. 3. 6. 1 (ref. 

2), states that the design review program should be defined and submitted as part of the Reliabil- 
ity Program Plan. The review plan will contain a schedule of reviews (contractor and subcon- 
tractor reviews as applicable) by level and type. Certain generalizations can be accepted in the 
initial listings since all major components may not be identified, and initial schedules will be 
keyed to milestones. It is recognized that the principles of the design review process should be 
applied in different degrees to the various types of NASA contracts. In order to provide a basis 
for formulation of effective design review programs for the large variety of NASA programs, the 
following questions should be answered for the particular program: 

(1) How important is the design review program to the achievement of reliability? 

(2) Why does a formal review program achieve anything? 

(3) How important are the assurance aspects of the design review program? 

An adequate program can be planned once these questions are resolved. Chapter 5 provides 
a technique for estimating costs of the selected review plan. The design review program is a 
product assurance program and assists in the improvement of marginal designs through the early 
visibility given the design logic. It is the only technique for independent evaluation prior to the 
test program. Since problems found at a later date tend to be "fixed" rather than corrected, the 
importance of design review is evident. 

The application of additional brainpower and experience to the design and development effort 
through design review enhances management visibility and permits more knowledgeable direction 
at the major design decision points. The result is product improvement. 

The formal program provides timely assurance to contractor and customer management of 
the validity of technical decisions. The review disciplines of retention and presentation of deci- 
sion logic and design data assure earlier attention to problem areas. The design review program 
is the principal technique which provides assurance to company and customer management prior 
to test and usage that the specification and the demonstrated capability of the product do satisfy 
the output criteria. 

With this background, it is now possible to discuss applicability of a design review program 
in terms of program requirements. A program of national significance, such as a manned space- 
craft, a man-rated launch vehicle, or a major space probe of the Surveyor or Voyager class, will 
require a complete design review program (both contractor and subcontractors), incorporating 
all the elements described in the preceding chapters . The program requirements for achievement 
of reliability and for assurance to company and customer management are very rigid, and any 
benefits which can be gained from the review program must be gained. The following listing 
illustrates that program: 

(1) System preliminary design review (PDR) 

(2) Subsystem PDR 

Guidance 
Flight controls 
Environmental control 
C ommunications 
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Range safety 

Electrical 

Hydraulic 

Malfunction detection system 

AGE 

Structure 

Etc. 

(3) Component PDR 

All major components 

(4) Component PDR— Subcontractor 

All major procured components 

(5) Subsystem status reviews 

All subsystems at 6-month intervals to first manned flight 

(6) Component prepackage reviews 

All major electronic and electromechanical components 

(7) Component prerelease reviews 

All major components 

(8) Component postqualification reviews 

All components in test program 

(9) Acceptance review 

All end items 

(10) Flight readiness reviews 

Prior to each flight 

Chapter 5 presents the visible costs of full application of those requirements of two repre- 
sentative programs, a man- rated launch vehicle and a lunar hard-landing probe. 

Programs of lesser complexity or significance can have review programs scaled to the in- 
creased inherent visibility of the less complex item. A relatively simple scientific experiment 
designed to be launched piggyback from another system can possibly be covered with a conceptual 
review, a prerelease review, and a flight- readiness review. 

There are, of course, many possible variations between these two extremes, dependent upon 
the degree of reliability achievement desired and the assurance required by the customer. The 
following list presents a review program plan for a hypothetical unmanned satellite system or 
new launch vehicle stage: 

(1) System PDR 

(2) Component PDR 

Major components 

(3) Component prerelease review 

Newly designed major components 

(4) Acceptance review 

End items 

(5) Flight-readiness review 

End items 

The requirements of NPC 250-1 (ref. 2) are applied at a level commensurate with the design chal- 
lenge and the significance and expense of the program. In this case, the maximum review pro- 
gram is not required and conceptual reviews are limited to system and major components. Pre- 
release reviews are held on major components, and an acceptance review and a flight- readiness 
review are held on the end item. Chapter 5 presents a cislunar micrometeoroid detector as an 
example of this class of program. 
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Similarly, a program utilizing proven hardware for a new purpose (e. g. , man-rating Atlas D 
for Mercury use). may require heavy emphasis on the assurance aspects of the design review pro- 
gram with greatly reduced emphasis on design change to improve the reliability of the hardware. 


CHAPTER 5 


Costs 

Application of the precepts of this manual can be expected to result in a reduction of end costs 
of any program through prudent and timely investment of manpower to reduce the larger cost risks 
of subsequent failure. However, it is equally necessary to provide guidance for use in making 
initial program cost estimates for the apparent costs of design review programs. 

Frequently we find that the documentation disciplines necessary for technical control and sub- 
stantiation of a development program have been neglected. It then becomes quite costly to sub- 
stantiate the development after the fact; however, such costs should not be chargeable to the de- 
sign review program costs. It has been mentioned before that one of the major benefits of the 
review program for the achievement of reliability is derived from the documentation disciplines 
which are necessary to substantiate the development effort. Any program having a valid reliabil- 
ity criterion (either quantitative or qualitative) should be sufficiently well documented that no 
efforts other than collection and distribution of existing documents (not just data) are necessary. 
The review program costs derived herein presume that this discipline has been exercised. 

In order to develop an estimate of apparent costs we have utilized a review program format 
based on the ground rules contained in the earlier chapters of this manual. Starting with individual 
reviews (system preliminary, component prerelease, etc.), we have established manpower com- 
plements which by virtue of their specialized talents are believed capable of accomplishing the re- 
view task. Tables 1 to 24 present complements of reviewers and other contributors along with 
estimated preparation and meeting time allotments . These individual complements may be applied 
to programs of varying complexity and national significance to gain some measure of the visible 
costs of design review in proportion to overall engineering effort. 

EXAMPLES OF PROGRAMS 
Mon-Rated Launch Vehicle 

A man- rated launch vehicle can be considered as an example of a program of national signifi- 
cance, as mentioned in chapter 4. The design and development, exclusive of the engines, could 
cost approximately $200 million if the design were just now being proposed. The engineering 
proposal should include design review as a separate task item (as part of the Reliability Program 
Plan) with an estimated component breakdown and design review plan time-phased with the engi- 
neering effort. Tables 25 and 26 present part of the information to be included in a similar pro- 
posal of a program of this magnitude. Further schedule presentations may be employed to inte- 
grate the element of review frequency into the proposal. This presentation might be visualized as 
a chart of system, subsystem, and component listings along one axis with program time along the 
other axis; each component listing would show estimated checkpoints where design reviews are 
planned. Or it might be simply a list identifying the system components as to complexity, type 
(electrical, electromechanical, or mechanical), and development status including prior use or 
new use. 

Application of the individual man-hour totals as estimated in tables 1 to 24 results in a re- 
view program effort totaling some 44, 800 man-hours. This estimate amounts to about 1. 1 per- 
cent of the total engineering manpower required for the definition and acquisition phases. 

Major Space Probe 

A major space probe of relatively high complexity and national significance is selected as 
another example. As in the case of the manned launch vehicle, full application of the proposed 
review techniques was suggested. 
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Tables 27 and 28 present the component breakdown and program of reviews which represent 
about 24, 600 man-hours. A comparison with total proposed engineering man-hours is not 
available . 

Several distinct features of this example should be mentioned. First, the above effort is 
limited to a late configuration. Additional review effort would have to be apportioned for the de- 
sign of the lunar survival package and additional sensors that were a part of earlier configura- 
tions. Second, both the spacecraft "bus” and the television system components are included in 
the example. Roughly 20 percent of the total manpower estimate would be performed by the tele- 
vision system subcontractor. Third, the Central Computer and Sequencer (CC&S) package is too 
complex to be handled as a component and would probably be best handled as a subsystem by 
frequent status reviews. Also the power subsystem consists mainly of two types of component 
(solar panels and storage batteries) and would probably not require subsystem reviews. The six 
subsystems indicated are: 

(1) Radio, data encoder, and command subsystem 

(2) Central computer and sequencer subsystem 

(3) Attitude-control subsystem 

(4) Midcourse propulsion subsystem 

(5) Television subsystem 

(6) Group support and data recovery 

Unmanned Satellite System 

A design review program of lesser proportions would probably be utilized in the development 
of an unmanned satellite system, say a cislunar micrometeoroid detector. Assuming a design 
aimed primarily at obtaining penetration data (but also recording and relaying secondary mea- 
surements), we have a program value in the $20 to $25 million range for the satellite and test 
phases, while the total program value might reach $40 million. Tables 29 and 30 present the 
design review effort suggested. The major components (average and high complexity) can be 
adequately covered with two reviews, preliminary and prerelease. The system level effort con- 
sists of a preliminary review, an acceptance review, and a flight-readiness review. A factor 
compensating for the reduction in review coverage is added inherent visibility of the more 
compact working force. 

This amounts to a design review program effort of 8500 man-hours. By utilizing a fairly 
conservative estimate of 500, 000 man-hours for total engineering effort to develop the satellite 
and AGE checkout equipment, the review program amounts to about 1. 7 percent of that total. 

COMPARISON WITH INDUSTRY PRACTICES 

By way of comparison, an Aerospace Industries Association survey of industry design re- 
view practices (ref. 15) showed between 1 and 28 percent of design man-hours were expended for 
this purpose, with a typical figure of 5 percent. If it is assumed that design effort is about 35 to 
50 percent of total engineering effort (the remainder being test site activation, test, systems inte- 
gration, reliability, quality assurance, program planning and control, fabrication, etc.), the 
typical figure given above falls between 1. 7 and 2. 5 percent. We have presented the examples in 
terms of total engineering effort since that reference has more validity for a design review pro- 
gram which must evaluate not only design but assurance efforts also. 

The conclusion to be reached from this discussion is that, in costing a design review pro- 
gram, project management must adapt the review effort to program complexity and significance, 
with the actual manpower allocation being about 1 or 2 percent of the total engineering effort. 
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Table 1. —System Preliminary Review— High Complexity 



Preparation, 

hr 

Meeting, 

hr 

Total, 

hr 

Review team 




Chief engineer (contractor): chairman 

88 

32 

120 

Sr. design specialists (3) 

144 

96 

240 

Design specialist (1 per subsystem; 10 subsystems). . 

480 

320 

800 

Staff reliability engineer 

48 

32 

80 

Customer systems engineer 

■ • • 

... 

... 

Customer design and test specialists (3) 



1240 

Other participants 




System engineer (technical director) 

0 

32 

32 

Subsystem design (1 per subsystem) 

0 

320 

320 

Reliability manager 

0 

32 

32 

Maintainability 

0 

32 

32 

Safety 

0 

32 

32 

Human factors 

0 

32 

32 

Value 

0 

32 

32 

Quality control 

48 

32 

80 

Manufacturing 

48 

32 

80 

672 

1912 


Table 2,— System Preliminary Review— Average Complexity 



Preparation, 

hr 

Meeting, 

hr 

Total, 

hr 

Review team 

Chief engineer (contractor): chairman 

64 

28 

92 

Staff reliability engineer 

36 

28 

64 

Sr. design specialist (2) 

72 

56 

128 

Design specialist (1 per subsystem; 7 subsystems) . . 

252 

196 

448 

Customer system engineer 

. . . 

. • • 

• • . 

Customer design and test specialists (3) 

. . . 

• • • 

• . • 




732 

Other participants 

System engineer (technical director) 

0 

28 

28 

Subsystem design (1 per subsystem) 

0 

196 

196 

Reliability manager 

0 

28 

28 

Maintainability. 

0 

28 

28 

Safety. 

0 

28 

28 

Value 

0 

28 

28 

Human factors 

0 

28 

28 

Oliolit’ir onnfrAl 

36 

28 

64 

Manufacturing 

36 

28 

64 




492 




1224 
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Table 3.— System Preliminary Review— Low Complexity 



Preparation, 

hr 

Meeting, 

hr 

Total, 

hr 

Review team 




Chief engineer (contractor): chairman 

48 

24 

72 

Design specialist (1 per subsystem; 4 subsystems). . . 

144 

96 

240 

Staff reliability engineer 

36 

24 

60 

Customer system engineer. 

. . . 



Customer design specialists (2) 

. . . 

• • • 

• • • 




372 

Other participants 




System engineer (technical director) 

0 

24 

24 

Subsystem design (1 per subsystem) 

0 

96 

96 

Reliability manager 

0 

24 

24 

S^fftty. . t , t _ t 

0 

24 

24 

Quality control 

36 

24 

60 

Manufacturing 

36 

24 

60 




288 

660 


Table 4.— Subsystem Preliminary Review (Airborne) 



Preparation, 

hr 

Meeting , 
hr 

Total , 
hr 

Review team 




Design specialist: chairman 

48 

16 

64 

Design specialist 

24 

16 

40 

Sr. Design specialists (2) 

48 

32 

80 

Staff reliability engineer 

24 

16 

40 

Circuit specialist 

24 

16 

40 

264 

Other participants 




Subsystem design 

0 

16 

16 

Reliability engineer 

0 

16 

16 

Maintainability. 

0 

16 

16 

Human factors 

0 

16 

16 

V alue 

0 

16 

16 

Quality control 

16 

16 

32 

Customer design and test specialists (2) 



112 

376 
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Table 5.— Subsystem Preliminary Review (Age) 



Preparation, 

hr 

Meeting, 

hr 

Total, 

hr 

Review team 

Design specialist: chairman 

48 

16 

64 

Design specialist 

24 

16 

40 

Sr. design specialists (2) 

48 

32 

80 

Staff reliability engineer 

24 

16 

40 

Circuit specialist 

24 

16 

40 




264 

Other participants 

Subsystem design 

0 

16 

16 

Reliability engineer 

0 

16 

16 

Maintainability 

0 

16 

16 

Human factors 

0 

16 

16 

Safety. 

0 

16 

16 

Quality control 

16 

16 

32 

Customer design and test specialists (2) 

• • . 

• * • 

• • . 




112 




376 


Table 6. —Subsystem Status Review (Airborne) 



Preparation, 

hr 

Meeting, 

hr 

Total, 

hr 

Review team 

Design specialist: chairman 

32 

12 

44 

Sr. design specialist 

32 

24 

56 

Staff reliability engineer 

16 

12 

28 

Circuit specialist 

16 

12 

28 




156 

Other participants 

Subsystem design 

0 

12 

12 

Reliability engineer 

0 

12 

12 

Maintainability 

0 

12 

12 

Value 

0 

12 

12 

Quality control 

12 

12 

24 

Customer design and test specialists (2) 

. . . 

. . . 

• • • 

72 




“228 
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Table 7. —Subsystem Status Review (Age) 



Preparation, 

hr 

Meeting, 

hr 

Total, 

hr 

Review team 




Design specialist: chairman 

32 

12 

44 

Sr. design specialist 

32 

24 

56 

Staff reliability engineer 

16 

12 

28 

Circuit specialist 

16 

12 

28 

156 

Other participants 




Subsystem design. 

0 

12 

12 

Reliability engineer 

0 

12 

12 

Maintainability 

0 

12 

12 

Safety 

0 

12 

12 

Quality control 

12 

12 

24 

Customer design and test specialists (2). 



72 

228 


Table 8. —Component Preliminary Review— High Complexity 



Preparation, 

hr 



Meeting, 

hr 

Total, 

hr 

Review team 

Design specialist: chairman 

28 

12 

40 

Sr. design specialists (2) 

24 

24 

48 




88 

Other participants 

Subsystem design 

0 

12 

12 

Component design 

0 

12 

12 

Reliability engineer 

0 

12 

12 

Human factors 

0 

12 

12 

Maintainability 

0 

12 

12 

Value 

0 

12 

12 

Quality control 

12 

12 

24 

Manufacturing 

12 

12 

24 

Customer design specialist 

. • . 

. . . 

... 




120 




208 
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Table 9. —Component Preliminary Review —Aver age Complexity 



Preparation, 

hr 

Meeting, 

hr 

Total, 

hr 

Review team 




Design specialist: chairman 

24 

8 

32 

Sr. design specialist 

8 

8 

16 

Design specialist 

8 

8 

16 

64 

Other participants 




Subsystem design 

0 

8 

8 

Component design 

0 

8 

8 

Reliability engineer 

0 

8 

8 

Quality control 

8 

8 

16 

Manufacturing 

8 

8 

16 

Customer design specialist 

... 

... 

56 

120 


Table 10. —Component Preliminary Review— Low Complexity 



Preparation, 

hr 

Meeting, 

hr 

Total, 

hr 

Review team 




Design specialist: chairman 

12 

4 

16 

Design specialist 

4 

4 

8 

Reliability engineer 

4 

4 

8 

32 

Other participants 




Component design 

0 

4 

4 

Customer design specialist 



4 

36 
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Table 11. —Component Prepackaging Review— High Complexity 
(Electronic or Electromechanical Designs Only) 



Preparation, 

hr 

Meeting, 

hr 

Total, 

hr 

Review team 




Design specialist: chairman 

32 

8 

40 

Packaging specialist 

8 

8 

16 

Circuit specialist 

16 

8 

24 

Staff reliability engineer 

16 

8 

24 

104 

Other participants 




Subsystem design 

0 

8 

8 

Component design (circuits) 

0 

8 

8 

Component design (packaging) 

0 

8 

8 

Maintainability 

0 

8 

8 

Customer design specialist 

. . . 

. . . 

• . . 


32 


136 


Table 12. —Component Prepackaging Review— Average Complexity 
(Electronic or Electromechanical Designs Only) 



Preparation, 

Meeting, 

Total , 


hr 

hr 

hr 

Review team 

Design specialist: Chairman 

26 

6 

32 

Circuit specialist 

10 

6 

16 

Packaging specialist 

10 

6 

16 

Reliability engineer 

10 

6 

16 


i — - — 


80 

Other participants 

Component design (circuits) 

0 

6 

6 

Component design (packaging) 

0 

6 

6 




12 




92 
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Table 13 . —Component Prerelease Review— High Complexity 
(Electronic or Electromechanical Designs Only) 



Preparation, 

hr 

Meeting, 

hr 

Total, 

hr 

Review team 




Design specialist: chairman 

32 

8 

40 

Circuit specialist 

8 

8 

16 

Packaging specialist 

8 

8 

16 

Staff reliability engineer 

16 

8 

24 

Environments specialist 

8 

8 

16 

Manufacturing 

16 

8 

24 

136 

Other participants 




Subsystem design 

0 

8 

8 

Component design (packaging) 

0 

8 

8 

Maintainability 

0 

8 

8 

Quality control 

8 

8 

16 

40 

176 


Table 14. —Component Prerelease Review— High Complexity 
(Mechanical Designs Only) 



Preparation, 

hr 

Meeting, 

hr 

Total, 
hr 1 

Review team 

Design specialist: chairman 

32 

8 

40 

Packaging specialist 

8 

8 

16 

Staff reliability engineer 

16 

8 

24 

Environments specialist 

8 

8 

16 

Manufacturing 

16 

8 

24 




120 

Other participants 

Subsystem design 

0 

8 

8 

Component design (packaging) 

0 

8 

8 

Maintainability 

0 

8 

8 

Quality control 

8 

8 

16 




40 




160 
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Table 15. —Component Prerelease Review— Average Complexity 



Preparation, 

hr 

Meeting, 

hr 

Total, 

hr 

Review team 




Design specialist: chairman 

28 

4 

32 

Reliability engineer 

12 

4 

16 

Environments specialist 

12 

4 

16 

Manufacturing 

12 

4 

16 

80 

Other participants 




Subsystem design 

0 

4 

4 

Component design (packaging) 

0 

4 

4 

Maintainability 

0 

4 

4 

Quality control 

12 

4 

j 

16 

28 

108 


Table 16 . —Component Postqualification Review— High Complexity 



Preparation, 

hr 

Meeting, 

hr 

Total, 

hr 

Review team 




Design specialist: chairman 

18 

6 

24 

Staff reliability engineer 

10 

6 

16 

Environments specialist 

10 

6 

16 

Quality control 

10 

6 

16 

72 

Other participants 




Component design 

0 

6 

6 

Test conductor 

0 

6 

6 

Manufacturing * 

10 

6 

16 

28 

100 
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Table 17. —Component Postqualification Review— Average Complexity 



Preparation, 

hr 

Meeting, 

hr 

Total, 

hr 

Review team 




Design specialist: chairman 

12 

4 

16 

Reliability engineer 

4 

4 

8 

Environments specialist 

4 

4 

8 

Quality control 

4 

4 

8 

40 

Other participants 




Component design 

0 

4 

4 

Test conductor 

0 

4 

4 

Manufacturing 

4 

4 

8 

16 

56 


Table 18 . —Component Postqualification Review— Low Complexity 



Preparation, 

hr 

Meeting, 

hr 

Total, 

hr 

Review team 




Design specialist: chairman 

6 

2 

8 

Environments specialist 

2 

2 

4 




12 

Other participants 




Component design 

0 

2 

2 

Test conductor. 

0 

2 

2 




4 

16 
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Table 19. —Combined Prerelease/Postqualification Review— High 


Complexity (Prior Usage Components) 



Preparation, 

hr 

Meeting, 

hr 

Total, 

hr 

Review team 




Design specialist: chairman 

40 

8 

48 

Design specialist 

16 

8 

24 

Staff reliability engineer 

14 

8 

22 

Environments specialist 

14 

8 

22 

Quality control 

14 

8 

22 

138 

Other participants 




Subsystem design 

0 

8 

8 

Component design 

0 

8 

8 

Maintainability. 

0 

8 

8 

Reliability engineer 

0 

8 

8 

Manufacturing 

14 

8 

22 

54 


Table 20.— Combined Prerelease/Postqualification Review— Average 
Complexity (Prior Usage Components) 



Preparation, 

hr 

Meeting , 
hr 

Total, 

hr 

Review team 




Design specialist: chairman 

34 

6 

40 

Reliability engineer 

12 

6 

18 

Environments specialist 

12 

6 

18 

Quality control 

12 

6 

18 

94 

Other participants 




Subsystem design 

0 

6 

6 

Component design 

0 

6 

6 

Maintainability. 

0 

6 

6 

Manufacturing 

12 

6 

18 

36 

130 
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Table 21. —Preacceptance Review (Contractor Preparation) 
[Average-3 Hours Per Subsystem, N Subsystems] 


Conducted by program manager 

Preparation, hr 
3N 

Other participants 

System engineer (technical director) 

3N 

Project engineers (reliability, test, safety, design) 

12N 

Quality control manager 

3N 

Subsystem design 

3N 

Subsystem manufacturing representative 

3N 

Subsystem reliability representative 

3N 

Subsystem test representative 

3N 

Subsystem quality control representative 

3N 


36N 


Table 22. —Acceptance Review (Customer) 
[N Subsystems ] 



Preparation, 
^ 

Meeting, 

hr 

Total, 

hr 

Review team (customer) 

Customer design specialist: chairman 


• ♦ • 

• • • 

Customer design and test specialists (approx 4). . . . . 

CD 

• • • 

• • • 

Other participants (contractor) 

O 

C 

ert 



Program manager 

CX 

40 

40 

System engineer (technical director) 

<D 

40 

40 

Project engineers (reliability, test, safety, design). . 

O 

O 

160 

160 

Subsystem design 

Oj 

40N 

40N 

Manufacturing 

u 

40 

40 

Quality control manager 

Cm 

40 

40 



320 

+ 40N 
1 
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Table 23. —Preflight Readiness Review (Contractor Preparation) 
[Meeting Estimate— 16 Hr] 



Preparation, hr 

Review team 


Chief engineer: chairman 

16 

Program manager 

16 

Site manager 

16 

Other participants 

""" '48 

System engineer (technical director) 

16 

Project engineers (reliability, test, safety, design) 

64 

Subsystem design (N subsystems) 

16N 

Quality control manager 

16 

Manufacturing 

16 

Site test engineers (approx 4) . 

64 

Site quality control 

16 

Site safety. 

16 

208 + 16N 
256 + 16N 


Table 24. —Flight -Readiness Review (Customer) 
[Meeting Estimate— 16 Hr] 



Preparation, 

hr 

Meeting, 

hr 

Total, 

hr 

Review team (customer) 

Customer design specialist: chairman 

i • • 



Customer design and test specialists (approx 4) . . . . 

• • • 



Other participants (contractor) 

Chief engineer 


16 

16 

Program manager 

CO 

16 

16 

System engineer (technical director) 

£ 

16 

16 

Project engineers (reliability, test, safety, design). . 


64 

64 

Subsystem design (N subsystems) 

<D 

16N 

16N 

Manufacturing 

?-i 

16 

16 

Quality control manager 

•g) 

16 

16 

Site test engineers (approx. 4) 

r-H 

64 

64 

Site quality control 

0) 

u 

16 

16 

Site safety 

CU 

16 

16 

Site manager 


16 

16 



256 + 16N 
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21 

Table 25. —Man-Rated Launch Vehicle Component Breakdown 


Component 

Total 

Electronic or 
electromechanic 2 il 

Mechanical 

Prior use 

New 

Prior use 

New 

High complexity 

20 

9 

9 

2 

0 

Average complexity 

31 

16 

10 

5 

0 

Low complexity 

49 

17 

13 

17 

2 

Total 

100 

42 

32 

24 

2 


Based on component information contained in ref. 8. 


Table 26. -Man-Rated Launch Vehicle Design Review Program 


No. of reviews 

Review type 

1 

System preliminary review (high complexity) 

10 

Subsystem preliminary reviews 

30 

Subsystem status reviews 

1 

Acceptance review with prior internal review 

12 

Flight readiness reviews with prior internal reviews 

20 

High-complexity component preliminary reviews 

31 

Average -complexity component preliminary reviews 

49 

Low-complexity component preliminary reviews 

9 

High-complexity component prepackaging reviews 

10 

Average-complexity component prepackaging reviews 

9 

High-complexity component prerelease reviews (elec) 

0 

High-complexity component prerelease reviews (mech) 

10 

Average-complexity component prerelease reviews (all) 

11 

High-complexity component combined reviews (prior use) 

21 

Average-complexity component combined reviews (prior use) 

9 

High-complexity component postqualification reviews 

10 

Average complexity component postqualification reviews 

49 

Low -complexity component postqualification reviews 


a 

Table 27.— Major Space Probe Component Breakdown 


Component 

Total 

Electronic or electromechanical 

Mechanical 

High complexity 

12 

9 

3 

Average complexity 

17 

15 

2 

Low complexity 

17 

14 

3 

Total 

46 

38 

8 


^ased on component information contained in refs, 9-13, 
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Table 28. —Major Space Probe Design Review Program 


Number 

Prime contractor I Subcontractor 



Review type 


System preliminary review (high complexity) 

Subsystem preliminary reviews 
Subsystem status reviews 

Acceptance review with prior internal reviews 
Flight readiness reviews 

High-complexity component preliminary reviews 
Average -complexity component preliminary reviews 
Low -complexity component preliminary reviews 
High-complexity component prepackaging reviews 
Average -complexity component prepackaging reviews 
High -complexity component prerelease reviews (elec) 
High-complexity component prerelease reviews (mech) 
Average -complexity component prerelease reviews (all) 
High-complexity component postqualification reviews 
Aver age -complexity component postqualification reviews 
Low -complexity component postqualification reviews 


Table 29. —Unmanned Satellite System Component Breakdown 


Component 

Total 

Electronic or electromechanical 

Mechanical 

High complexity 

9 

9 

0 

Average complexity 

11 

10 

1 

Low complexity 

12 

7 

5 

Total 

32 

26 

6 


^ased on estimated component structure contained in ref. 14. 


Table 30.— Unmanned Satellite System Design Review Program 


Review type 


System preliminary review (average complexity) 
Component preliminary reviews (high complexity) 
Component preliminary reviews (average complexity) 
Component prerelease reviews (high complexity) 
Component prerelease reviews (average complexity) 
Acceptance review (approx 6 subsystems) 

Flight readiness review 























APPENDIX A 


Definitions 

The following definitions apply to the terms used in this publication. 

Acceptance : The act of an authorized representative of the Government by which the Government 
assents to its ownership of existing and identified articles or approves specific services 
rendered as partial or complete performance of the contract (ref. 3). 

Aerospace ground equipment : All equipment required on the ground to make an aerospace system 
operational in its intended environment (ref. 16). 

AGE : Aerospace ground equipment. 

Article : A unit of hardware or any portion thereof required by the contract (ref. 3). 

Associate contractor : The contractor who under direct contract to NASA perforins work excluded 
from the principal contract. The associate contractor is responsible to the principal contrac- 
tor for technical integration of the (sub) system and must coordinate technical developments 
and requirements in a timely and organized manner. The associate contractor is directly 
responsible to NASA for administrative and contractual matters (ref. 16). 

Baseline configuration : The documented and approved design concept or arrangement of compon- 
ents as established at a given point in the procurement cycle for systems or equipment (ref. 
16). 

Block diagram : A line drawing with block outlines to designate units or functional groups for 
general arrangement studies, functional explanation, product familiarization, etc. within a 
system, set, or item (ref. 16). 

Breadboard : An assembly of preliminary circuits or parts used to prove the feasibility of a de- 
vice, circuit, system, or principle without regard to the final configuration or packaging of 
the parts (ref. 16). 

Checklist : A list of procedures or items summarizing the activities required for an operator or 
technician in the performance of his duties; a condensed guide; an on-the-job supplement to 
more detailed job instructions (ref. 16). 

Chief engineer : That person in charge of all technical activity in an organization. 

Circuit: A group of elements or parts, connected and related so as to perform a specific function 
in a component, assembly, or system (ref. 16). 

Cislunar : Of or pertaining to space between the Earth and the orbit of the Moon or to a sphere of 
space centered on the Earth with a radius equal to the distance between the Earth and the 
Moon (ref. 16). 

Cognizant NASA installation : That major organizational unit of NASA which has direct technical 
and managerial responsibility for the system under contract (ref. 2). 

Component: A combination of parts, subassemblies, or assemblies, usually self-contained, 

which performs a distinctive function in the operation of the overall equipment; a "black box" 
(ref. 2). 

Conceptual phase : That period in the system life cycle which usually terminates with publication 
of a specific operational requirement (ref. 16). 

Configuration : The technical and physical description required to fabricate, test, accept, oper- 
ate, maintain, and logistically support systems or equipment (ref. 16). 

Configuration control : Systematic evaluation, coordination, approval, or disapproval of all 
changes to the baseline configuration (ref. 16). 

Consent to ship : Customer buy-off function prior to transporting end item to field. 

Consent to test : Customer buy -off function prior to major ground test or launch. 

Contract : The agreement formally executed* by the Government and the prime contractor which 
requires performance of services and/or delivery of a product at a cost to the Government, 
in accordance with terms and conditions set forth therein (ref. 2). 

Contractor : Any person, partnership, company, or corporation (or any combination of these) 
which is a party to a contract with the United States Government (ref. 2) . 

Critical path : That particular sequence of activities that has the greatest negative or least posi- 
tive activity slack (ref. 16). 

CTS: Consent to ship. 
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CTT : Consent to test. 

Design criteria : Standards upon which a design is based (ref. 16). 

Design review program : A systematized and disciplined application of the broad technical compe- 
tence oiHthe contractor and the customer to a product. The program is intended to improve 
the product and to provide assurance to contractor and customer management, by formalized 
documentation of the decision logic, that the most satisfactory design has been selected to 
meet the program requirements. 

Design specification ; A document prescribing criteria to be satisfied in designing a particular 
component, subsystem, or system (or part). Typical criteria include performance require- 
ments under specified environments, interface requirements , size, weight, ruggedness, safety 
margins, derating factors, and apportioned reliability goal (with definition of failure) (ref. 2). 

Designated representative : An individual (such as a NASA plant representative) , firm (such as 
an assessment contractor), or Government agency designated and authorized by NASA to 
perform a specific function(s) relative to the contractor's reliability effort; e.g. , monitor- 
ship, assessment, and design review participation and/or approval of certain documents or 
actions (ref. 2) 

Detail drawing: Delineates information to describe an item and shall include materials, finish, 
tolerances, and other requirements as applicable (ref. 16). 

Detail specification : A detail description of a particular model of an item prepared by the de- 
signer which cites all specific design and construction criteria (ref. 16). 

Development : The application of known techniques and principles to produce a desired result 
from the discoveries of research. In the development stage a device is visualized and its 
performance is anticipated. Development is characterized by deliberate planning, by ingenu- 
ity, and by synthesis of knowledge in many fields. The result of development is the creation 
of plans or models for a new device and the demonstration by test that the prototype of the 
device fulfills the objective of the development (ref. 16). 

Deviation : A specific authorization, granted before the fact, to depart from a particular require- 
ment of specifications or related documents (ref. 3). 

End item : A space system or any of its principal system or subsystem elements; for example, 
launch vehicle, spacecraft, ground support system, propulsion engine, or guidance system. 
Also, articles covered by major subcontracts where NPC 200-2 is invoked by the NASA in- 
stallation or by a system prime contractor. Also, articles which will be delivered direct to a 
Government installation or provided as GFP to a contractor (ref. 3). 

Feasibility study : The phase during which studies are made of a proposed item or technique to 
determine the degree to which it is practicable, advisable, and adaptable for the intended 
purpose (ref. 16). 

First article configuration inspection (FACI) : A formal technical review which establishes the 
product configuration baseline (ref. 5). 

GLV : Gemini launch vehicle. 

Installation drawing : Shows general configurations, attaching hardware, and information to lo- 
cate, position, and mount an item relative to fixed points and to other items (ref. 16). 

Integration : The process of assuring that the major elements of a program be conceived, de- 
signed, assembled, tested, operated, and documented in such a manner as to be compatible 
with each other and to satisfy the program objectives (ref. 16). 

Interface : The point or area where a relationship exists between two or more parts, systems, 
programs, persons, or procedures wherein physical and functional compatibility is required 
(ref. 16). 

Interface drawing : The engineering drawing which graphically or descriptively displays the con- 
ditions of the interface which exist between assemblies (ref. 16). 

Launch integrity team : Review team composed of high-level representatives of the prime con- 
tractor, or the integration contractor and the major associates, to review each Gemini Launch 
Vehicle prior to the customer flight-readiness review. 

Launch vehicle : The part of the space vehicle which furnishes the propulsion and guidance during 
the initial part of the trajectory to provide the prescribed velocity, position, and attitude 
required for injection into the desired trajectory. Launch vehicles are commonly called 
boosters and consist of two or more propulsive stages (ref. 16). 

LIT : Launch integrity team. 

Man-rated space vehicle : Space vehicles for manned flight which have achieved the standards of 
performance and reliability previously established as reasonably acceptable for this class of 
equipment (ref. 16). 
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Milestone : Any significant event in the design and development of a space system or in the asso- 
ciated reliability program which is used as a control point for measurement of progress and 
effectiveness or for planning or redirecting future effort. Reliability program milestones 
should be identified in the reliability program plan (ref. 2). 

Mission task : The specified purpose for which a device must perform (ref. 16). 

NASA : National Aeronautics and Space Administration. 

National Aeronautics and Space Administration: Civilian agency with research and development 
jurisdiction in aeronautical and space activities, except those activities peculiar to and pri- 
marily associated with the development of weapons systems and military operations (ref, 16). 

Nonstandard part : One for which no published standard or specification exists (ref. 16). 

NSRA : Nonstandard request authorization. 

Operational : Equipment for which all research and development has been completed with achieve- 
ment of performance objectives (ref. 16). 

Operational phase : The period from acceptance by the user of the first operating unit until dispo- 
sition of the system equipment. The operational phase overlaps the acquisition phase (ref. 

16). 

Part : One piece or two or more pieces joined together which are not normally subject to dis- 
assembly without destruction (ref. 2) . 

PERT : Program evaluation and review technique. 

Piggyback experiment : An experiment which rides along with the primary experiment on a space - 
available basis without interfering with the mission of the primary experiment (ref. 16). 

Prime contractor : A contractor with total system responsibility for the execution of work on 
contract to a Government agency. This includes all functional and administrative responsi- 
bilities necessary to satisfy contract requirements. Major programs can be established with 
separate prime contractors for essentially independent systems, but each will perform as a 
contractual entity with respect to the contracting agency (ref. 16). 

Program : A related series of undertakings designed to accomplish a broad scientific or technical 
goal. Attainment of such long-range goals may be accomplished by implementation of 
specific projects (ref. 16). 

Program evaluation and review technique : Method of charting events and obtaining predicted per- 
formance in accordance with a schedule (ref. 16). 

Project : A scheduled undertaking, within a program, which may involve the research and devel- 
opment, design, construction, and operation of system and associated hardware, or hard- 
ware only, to accomplish a scientific or technical objective (ref. 16). 

Qualification : Determination by a series of tests and examination of documents and processes 
that a part, component, subsystem, or system is capable of meeting performance require- 
ments prescribed in the purchase specification or other documents specifying the adequate 
performance capability for the item in question (ref. 2). 

Redundancy (of design) : The use of more than one means of accomplishing a given task or function 
where all must fail before there is an overall failure of the system (ref. 2). 

Schematic diagram: A diagrammatic drawing that shows function symbols with interconnections 
to illustrate circuit operation. It does not necessarily identify physical location of compon- 
ents or connections between them (ref. 16). 

Space system : A system of equipment consisting of launch vehicle(s), spacecraft, ground sup- 
port equipment, and test hardware, used in ground testing, launching, operating, and 
maintaining space vehicles or spacecraft (ref. 2). 

Subcontract : A formal contract between a prime contractor to the Government and another con- 
cern or individual(s) (first tier subcontract) requiring performance of services and/or deliv- 
ery of a product required in connection with the prime contract, or a similar contractual 
agreement between a first tier subcontractor and another concern (second tier) (ref. 2). 

Subcontractor : A concern or individual (s) entering into a contract with the prime contractor or 
with a subcontractor in a tier higher than his own for services or a product required for the 
prime contract or a subcontract (ref. 2). 

System : One of the principal functioning entities comprising the project hardware and related 
operational services within a project or flight mission. Ordinarily, a system is the first 
major subdivision of project work. Similarly, a subsystem is a major functioning entity 
within a system. (A system may also be an organized and disciplined approach to accom- 
plish a task; e.g. , a failure -reporting system.) (ref. 2). 
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Systems engineering : The process of applying science and technology to the study and planning of 
~ a system so that the relationships of various parts of the system and the utilization of various 
subsystems are fully established before designs are committed (ref. 16) . 

Vehicle acceptance test : System and subsystem test to ensure vehicle specification compliance 
before vehicle is accepted for flight use (ref. 16). 

Waiver : Granted use or acceptance of an article which does not meet specified requirements 
(ref. 3). 
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"The aeronautical and space activities of the United States shall be 
conducted so as to contribute . . . to the expansion of human knowl- 
edge of phenomena in the atmosphere and space. The Administration 
shall provide for the widest practicable and appropriate dissemination 
of information concerning its activities and the results thereof ” 

— National Aeronautics and Space Act of 1958 


NASA SCIENTIFIC AND TECHNICAL PUBLICATIONS 


TECHNICAL REPORTS: Scientific and technical information considered 
important, complete, and a lasting contribution to existing knowledge. 

TECHNICAL NOTES: Information less broad in scope but nevertheless of 
importance as a contribution to existing knowledge. 

TECHNICAL MEMORANDUMS: Information receiving limited distribu- 
tion because of preliminary data, security classification, or other reasons. 

CONTRACTOR REPORTS: Scientific and technical information generated 

under a NASA contract or grant and considered an important contribution to 
existing knowledge. 

TECHNICAL TRANSLATIONS: Information published in a foreign 
language considered to merit NASA distribution in English. 

SPECIAL PUBLICATIONS: Information derived from or of value to NASA 
activities. Publications include conference proceedings, monographs, data 
compilations, handbooks, sourcebooks, and special bibliographies. 

TECHNOLOGY UTILIZATION PUBLICATIONS: Information on tech- 
nology used by NASA that may be of particular interest in commercial and other 
non-aerospace applications. Publications include Tech Briefs, Technology 
Utilization Reports and Notes, and Technology Surveys. 


Details on the availability of these publications may be obtained from: 


SCIENTIFIC AND TECHNICAL INFORMATION DIVISION 

NATIONAL AERONAUTICS AND SPACE ADMINISTRATION 

Washington, D.C. 20546 



